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OPINION 


GOVERNMENT  TRANSFORMATION 
AND  STRUCTURAL  RIGIDITY: 
RCDBSI6NING  A  SERVICE 
ACQUISmOH  PROCESS 

Dr.NedKo<k 


Organizations  have  been  increasingly  pushed  into  a  fast  pace  of  change  by 
the  globalization  of  the  economy,  the  accelerated  technological  developments 
in  information  storage  and  retrieval,  and  the  emergence  of  knowledge  (as 
opposed  to  capital  goods)  as  the  main  asset  of  organizations.  Many  radical 
change  approaches  have  been  developed  to  ease  this  transition.  While  these 
approaches  sometimes  succeed,  in  most  cases  they  fail  miserably— a 
phenomenon  that  has  been  usually  blamed  on  poor  change  management. 
The  author  argues  that  structural  factors  are  also  to  be  blamed,  particularly 
process  rigidity  caused  by  highly  functional  heterogeneity,  fragmented 
expertise,  and  government  regulation.  This  point  Is  supported  by  the  analysis 
of  a  re-engineering  attempt  of  a  core  process  of  a  public  sector  organization 
in  Brazil.  The  author  proposes  a  simple  framework  to  identify  process  structural 
rigidity  in  public  sector  organizations,  and  provide  the  basis  to  understand 
how  structural  rigidity  can  oppose  radical  change. 


Organizational  processes  today  are 
markedly  different  than  they  were 
100  years  ago.  It  has  been  esti¬ 
mated  that  in  1 880  about  9  out  of  10  work¬ 
ers  produced  and  moved  tangible,  mate¬ 
rial  things.  In  the  mid-1990s  this  ratio  was 
down  to  one  out  of  five.  The  other  four 
out  of  five  workers  currently  produce  and 
deliver  intangible  products  such  as  infor¬ 
mation,  computer  software,  and  services 
(Drucker,  1993). 


The  accelerated  development  of  new 
technologies,  combined  with  the  increas¬ 
ing  globalization  of  the  economy,  has 
helped  shape  a  global  market  in  which 
organizations  can  have  access  to  tools  that 
make  their  processes  efficient  and  effec¬ 
tive  anywhere  in  the  world.  So  for  most 
products  generated  and  transferred  within 
and  between  organizations  today,  whose 
soft  elements  (i.e.,  information,  software, 
and  service)  can  be  delivered  virtually  in- 
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dependently  of  physical  distance,  fierce 
competition  on  a  global  basis  has  become 
commonplace. 

To  survive  in  such  an  environment,  sev¬ 
eral  organizations  have  had  to  become 
“virtual  organizations,”  in  the  sense  that 
they  have  come  to  chiefly  depend  on 
knowledge  and  process  flexibility  to  gen¬ 
erate  and  deliver  products  on  a  competi¬ 
tive  basis  (Davidow  and  Malone,  1992; 
Mowshowitz,  1997).  Capital  goods  are  no 
longer  such  a  strategic  advantage  to  orga¬ 
nizations  as  process-related  knowledge  is, 
a  reality  that  is  reflected  in  the  often  high 
market  valuation  of  knowledge  and  other 
intangible  assets  (such  as  computer  sys¬ 
tems)  as  opposed  to  material  organiza¬ 
tional  assets  such  as  production  machin¬ 
ery  and  factory  buildings  (Strassman, 
1996;  Toffler,  1991). 

Organizational  flexibility,  as  well  as  the 
accumulation  and  proper  deployment  of 
process-related  knowledge,  depend  on 
structural  characteristics  of  organizations 
such  as  departmental  and  functional  con¬ 
figuration,  hierarchical  levels,  and  infor¬ 
mation  access  and  flow  (Argyris,  1977; 
Redding  and  Catalanello,  1994;  Senge, 
1990).  Hence,  it  is  surprising  to  see  the 
current  focus  on  change  management  in 
process-based  organizational  transforma¬ 
tion  efforts  (Kettinger  and  Grover,  1995; 
Stoddard  and  Jarvenpaa,  1995),  and  the 
relative  lack  of  interest  on  structural  fac¬ 
tors  that  themselves  can  prevent  organi¬ 
zational  change  from  happening. 


Here  we  describe  a  re-engineering  at¬ 
tempt  in  a  public  sector  organization  in 
Brazil,  particularly  regarding  the  attempt 
to  redesign  one  of  its  core  processes — a 
service  acquisition  process.  This  provides 
the  basis  for  our  subsequent  discussion  of 
structural  factors  that  can  prevent  radical 
process-based  change  projects  from 
achieving  successful  results.  The  focus  of 
this  discussion  is  on  process  rigidity  in 
public  sector  organizations,  and  its  rela¬ 
tionship  with  process  functional  hetero¬ 
geneity  and  degree  of  government  regu¬ 
lation.  These  factors  are  defined  from  a 
process-centered  perspective,  and  their 
link  with  knowledge  specialization  is  es¬ 
tablished.  We  conclude  with  a  compari¬ 
son  of  our  case  and  other  cases  in  the  lit¬ 
erature,  and  derive  implications  for  pub¬ 
lic  sector  organizations  interested  in  (or 
already)  conducting  process-based  radical 
change  projects. 

Radical  Change's  Failure: 

BiAME  It  On  Bad  Mahagimemt! 

The  movement  in  favor  of  radical  pro- 
cess-based  change  leveraged  by  informa- 
tion  technology  known  as  re-engineering 
begun  in  1990  with  two  seminal  articles 
by  Thomas  Davenport  and  James  Short 
(1990),  and  Michael  Hammer  (1990). 
Having  gone  beyond  its  initial  phase  of 
optimism,  which  reached  its  peak  in  the 
two  years  following  the  publication  of  the 


Dr.  Ned  Kock  is  an  assistant  professor  in  the  Department  of  Computer  and  Information  Sci¬ 
ences.  Temple  University,  Philadelphia,  PA.  He  holds  a  B.E.E.  degree  in  electronics  engineer- 
ina  an  M  S.  degree  in  computer  science,  and  a  Ph.D.  degree  in  information  systems  from  the 
University  of  Waikato,  New  Zealand.  He  is  the  author  of  the  book  Process  ^e-engrneer/ng 
PRO/:  A  Practical  Methodology,  published  in  1 995  in  Brazil  by  Vozes,  and  recipient  of  the  MCB 
Press  Outstanding  Paper  Award  in  1997  in  York,  England. 
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book  Reengineering  the  Corporation 
authored  by  Hammer  and  Champy  (1993), 
the  re-engineering  movement  entered  a 
crisis  stage  (Deakins  and  Makgill,  1997). 
It  was  argued  that  re-engineering  had  led 
to  heightened  worker  stress  and  lack  of 
process  outcome  quality  in  a  number  of 
organizations  where  it  had  been  imple¬ 
mented  (Burke  and  Peppard,  1995;  Econo¬ 
mist,  1996;Labich,  1994;  Willmott,  1995). 
Disagreements  in  the  mid  1990s  between 
what  many  saw  as  the  main  forefathers  of 
the  re-engineering  movement,  Michael 
Hammer  and  James  Champy,  worsened 
the  crisis.  While  Hammer  has  continued 
to  focus  on  techniques  to  radically  im¬ 
prove  processes  (Hammer  and  Stanton, 
1995),  Champy  pointed  out  that  re-engi¬ 
neering  success  would  not  likely  be 
achieved  without  a  radical  change  in  man¬ 
agement  paradigms  (Champy,  1995). 

Much  has  been  published  about  reasons 
why  re-engineering  attempts  may  fail,  as 
well  as  about  success  factors  in  re-engi¬ 
neering  (both  types  of  account  highlight 
essentially  the  same,  as  success  factors  can 
be  seen  as  factors  whose  presence  is  likely 
to  prevent  a  re-engineering  attempt  from 
foundering).  The  reasons  provided  in  the 
literature  to  explain  why  re-engineering 
so  often  fails  have  had  a  common  focus — 
management.  The  literature  generally  sug¬ 
gests  that  re-engineering  attempts  are 
likely  to  fail  if  they  lack  top  management 
support,  which  is  often  cited  as  the  chief 
reason  for  failure  (Archer  and  Bowker, 
1995;  Hall  et  al.,  1993).  It  also  highlights 
the  importance  of  management  selection 
and  adoption  of  a  structured  methodology 
for  process  redesign  (Guha  et  al.,  1993; 
Wasted  et  al.,  1994).  Lack  of  understand¬ 
ing  by  management  of  what  re-engineer¬ 
ing  really  entails  is  also  cited  as  a  source 


of  failure  (Champy,  1995).  King  (1997) 
points  out  that  since  there  is  no  univer¬ 
sally  accepted  definition  of  organizational 
transformation,  a  notable  source  of  fail¬ 
ure  in  re-engineering  is  the  confusion  by 
management  of  incremental  with  radical 
process  improvement.  Top  management 
leadership  and 

strong  involve-  literature 

ment  is  singled  generally  suggests 
out  by  Hewitt  i;  that  re-engineering 
and  Yeon  s  attempts  are  likely 

(1996)  survey  they Jiiek 

of  United  King-  >“«"e9ement 

,  support,  which  is 

doi"  companies  as  the 

engaged  in  re-  oAef  reason  for 

engineering  at-  failure/' 
tempts  as  the 

main  success  factor  in  radical  process- 
based  organizational  redesign. 

Although  re-engineering  has  been  prac¬ 
ticed  in  a  variety  of  industries  and  eco¬ 
nomic  sectors  (Hewitt  and  Yeon’s,  1996) 
with  both  positive  (Bell,  1994;  Caron  et 
al.,  1994)  and  negative  results  (Champy, 
1995),  little  has  been  said  about  the  influ¬ 
ence  of  structural  factors  (i.e.,  those  re¬ 
lated  to  an  organization’s  structure)  on  re¬ 
engineering  success  and  failure.  Can  suc¬ 
cess  factors  be  contingent  on  specific  in¬ 
dustry  or  economic  sector  characteristics? 
Or,  more  specifically,  can  certain  charac¬ 
teristics  of  a  specific  industry  or  sector  of 
the  economy  influence  the  structural  ri¬ 
gidity  of  organizational  processes — i.e. 
their  structural  resistance  to  change — so 
as  to  make  re-engineering  more  likely  to 
fail?  The  case  study-based  research  dis¬ 
cussed  here  suggests  a  positive  answer  to 
this  question  as  regards  one  particular  sec¬ 
tor  of  the  economy — the  public  sector.  The 
case  study  on  which  the  research  was 
based  is  described  next. 
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Case  Study:  Re-engineering  a 
Service  Acquisition  Process 


Since  the  early  1990s,  corporate  Brazil 
experienced  a  growing  interest  in  the 
adoption  of  voluntary  quality  standards; 
interest  that  was  fed  by  a  number  of  suc¬ 
cess  stories  (Ottoni,  1993).  The  most 
popular  among  these  quality  standards 
have  been  the  ISO  9002,  adopted  by  com¬ 
panies  that  produced  goods  or  services 
based  on  third-party  specifications;  the 
ISO  9001,  adopted  by  companies  which 
developed  their  own  products  and  ser¬ 
vices;  and  several  versions  of  these  two 
standards  tailored  for  specific  sectors  and 
industries  (Arnold,  1994;  Voehl  et  al,, 


1994). 

On  the  tail  of  the  success  of  such  ori¬ 
entation  toward  voluntary  standards  came 

a  growing  un¬ 
certainty  about 
the  need  for 
government 
regulatory  bod¬ 
ies  and  state- 
owned  inspec¬ 
tion  companies 
in  a  number  of 
industries.  One 
such  company 
was  PubliCorp 
(pseudonym),  a  large  civil  engineering  and 
construction  services  inspection  company 
owned  by  a  state  government  in  Brazil. 
Among  PubliCorp’s  main  missions  was 
that  of  enforcing  government  regulations 


"Pressed  to  become 
more  competitive, 
PubliCorp  set  out  in 
1992  to  re-engineer 
several  of  its  busi¬ 
ness  processes, 
including  one  of  its 
core  processes— the 
acquisition  of  con¬ 
struction  services." 


in  the  construction  industry. 

The  prospect  of  deregulation  pushed 
PubliCorp  into  considering  moving  from 
an  enforcement  role  to,  possibly,  a  qual¬ 
ity  consulting  one.  This  would  also  force 
PubliCorp  into  a  situation  in  which  it 


would  have  to  supply  added-value  services 
to  construction  companies,  as  it  would  no 
longer  be  able  to  deliver  compulsory-pur¬ 
chase  inspection  services.  As  a  conse¬ 
quence,  PubliCorp’s  future  survival  would 
depend  on  the  efficiency  of  its  processes 
and  the  quality  of  the  services  delivered 
through  them. 

Pressed  to  become  more  competitive, 
PubliCorp  set  out  in  1992  to  re-engineer 
several  of  its  business  processes,  includ¬ 
ing  one  of  its  core  processes — the  acqui¬ 
sition  of  construction  services.  As 
PubliCorp  was  a  state-owned  enterprise, 
the  acquisition  of  services  had  to  be  made 
through  the  setting  up  and  coordination 
of  public  bids,  whereby  PubliCorp  was 
expected  to  select  the  most  competitive 
contractors  to  carry  out  construction  and 
maintenance  jobs  on  public  estate  and 
transportation  networks.  This  re-engineer¬ 
ing  project  was  seen  by  PubliCorp’s  top 
management  and  government  officials  as 
a  test  of  the  company’s  ability  to  compete 
in  what  some  of  them  believed  could  in 
the  future  be  a  largely  deregulated  civil 
construction  industry. 

The  Stages  of  the 
Re-engineering  Ahempt 

The  research  and  normative  literatures 
on  business  process  re-engineering  have 
identified  a  number  of  generic  features  that 
seem  to  be  present  in  re-engineering  at¬ 
tempts,  whether  these  attempts  fail  or  suc¬ 
ceed  to  deliver  the  expected  outcomes. 
Two  of  these  generic  features,  both  present 
in  the  re-engineering  attempt  at  PubliCorp, 
are  (Davenport  and  Stoddard,  1994):  First 
is  a  focus  on  core  processes  that  involve 
several  departments  or  the  whole  organi¬ 
zation.  Core  processes  are  defined  (Kock 
et  al.,  1997)  as  those  processes  related  to 
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the  production  and  delivery  of  outputs  to 
the  external  customers  of  the  organization. 
Second  is  the  use  of  information  technol¬ 
ogy  (IT)  to  enable  the  implementation  of 
the  new  business  processes  devised 
through  the  re-engineering  effort. 

Another  unfortunate  characteristic  of 
most  re-engineering  attempts  has  been  a 
consistent  failure  to  deliver  the  expected 
outcomes,  of  which  the  most  important 
are  a  radical  improvement  in  the  effi¬ 
ciency  of  the  processes  redesigned  or  of 
the  customer-perceived  quality  of  the  out¬ 
puts  of  those  processes.  As  far  as  failure 
rates  of  re-engineering  attempts  go,  a 
widely  cited  figure  is  that  obtained  in  a 
survey  discussed  by  Champy  (1995): 
Seventy  percent  or  more  of  all  re-engi¬ 
neering  attempts  fail  to  attain  their 
goals.  In  this  respect,  the  re-engineer¬ 
ing  attempt  at  PubliCorp  was  also  typi¬ 
cal  in  that  it  too  failed  to  achieve  its  goals. 
The  dynamics  of  this  failure  can  be  more 
easily  understood  by  splitting  the  attempt 
into  five  main  stages:  problem  defini¬ 
tion  and  planning,  IT  infrastructure 
implementation,  IT  do wnsizing,  core  pro¬ 
cess  re-engineering  attempts,  and  process 
automation. 

Stage  1:  Problem  definition  and  plan¬ 
ning,  Two  small  work  groups  with  10  to 
12  members  each  (the  number  of  mem¬ 
bers  varied  slightly  along  the  whole  at¬ 
tempt)  were  formed  to  tackle  different  is¬ 
sues  in  the  re-engineering  attempt.  The  IT 
group’s  main  goal  was  to  deal  with  the 
technical  issues  related  to  the  setting  up 
of  an  IT  infrastructure  to  enable  the  re¬ 
engineered  processes.  The  process  rede¬ 
sign  (PR)  group  was  assigned  the  role  of 
analyzing,  proposing  radical  changes  in  the 
target  business  processes,  and  coordinating 
the  implementation  of  these  changes. 


Re-engineering  projects  often  start  with 
the  identification  of  urgent  problems  that 
are  expected  to  be  solved  through  radical 
process  redesign  (Hammer,  1996;  Ham¬ 
mer  and  Champy,  1993).  Two  such  prob¬ 
lems  were  identified  at  PubliCorp  regard¬ 
ing  the  process  of  acquisition  of  construc¬ 
tion  services:  First  was  the  centralization 
of  data  processing  jobs  in  one  department, 
largely  due  to 

the  fact  that  "Another  unforiu- 
product  and  ser-  note  characteristic  of 
vice  supplier  most  re-engineering 
databases  were  attempts  has  been  a 
kept  in  a  central  consistent  failure  to 
mainframe  deliver  the  expected 
computer  oper-  es...." 

ated  by  that  de¬ 
partment;  second  was  the  large  number  of 
“contact  points”  in  the  acquisition  process, 
caused  mainly  by  a  disproportionate  num¬ 
ber  of  specialized  tasks  and  control  checks 
that  had  to  be  performed  by  employees 
with  expertise  in  different  areas  (e.g.,  tax 
lawyers,  construction  budget  experts,  en¬ 
gineers  specializing  in  building  structures, 
concrete  experts,  public  bid  advisers).  A 
re-engineering  project  schedule  was  laid 
out  to  address  these  problems,  which  set 
up  a  number  of  steps  for  IT  infrastruc¬ 
ture  development  as  well  as  business  pro¬ 
cess  analysis,  radical  redesign,  piloting, 
and  incorporation  into  the  organizational 
structure. 

Stage  2:  IT  infrastructure  implemen¬ 
tation,  Solutions  to  both  problems  identi¬ 
fied  in  Stage  1  of  the  re-engineering  at¬ 
tempt  were  seen  by  the  IT  and  PR  groups 
as  begging  a  new  IT  infrastructure  based 
on  a  local  area  network  (LAN),  which  was 
expected  to  produce  immediate  gains  on 
its  own,  as  well  as  support  the  implemen¬ 
tation  of  changes  in  the  core  acquisition 
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process.  In  approximately  eight  months 
three  LAN  servers  with  over  100  net¬ 
worked  workstations  had  been  set  up.  This 
set  the  stage  for  the  downsizing  of  data¬ 
bases  and  data  processing  applications 
from  the  central  mainframe  computer  to 
the  LAN  server. 

Stage  3:  IT  downsizing.  The  down-siz¬ 
ing  of  database  applications  from 
PubliCorp’s  central  mainframe  computer 
to  the  LAN  server  was  seen  by  manage¬ 
ment  and  employees  as  having  itself  in¬ 
creased  the  efficiency  of  the  core  acquisi¬ 
tion  process  targeted  for  redesign,  by  al¬ 
lowing  fast  and  decentralized  access  by 
all  staff  involved  in  the  process  to  prod¬ 
uct  and  service  supplier  information,  as 
well  as  supporting  the  implementation  of 
process  simplifications  aimed  at  reducing 
“contact  points”  between  staff. 

Although  some  technical  problems  and 
opposition  from  the  old  centralized  data 
processing  department  had  to  be  dealt 
with,  everything  seemed  to  be  going  as 
planned  and  nothing  suggested  that  the  re¬ 
engineering  attempt  was  not  in  its  way 
toward  a  successful  completion.  The  gen¬ 
eral  feeling  among  management  at  this 
stage  was  that  there  was  only  one  type  of 
obstacle  to  be  overcome  so  the  re-engi¬ 
neering  attempt  would  eventually  succeed: 
technical  obstacles.  However,  nearly  two 
years  had  gone  by  since  the  initial  deci¬ 
sion  to  conduct  the  re-engineering  attempt 
had  been  made,  and  yet  no  actual  changes 
of  radical  proportions  in  any  of  PubliCorp’s 
core  processes  had  been  effected. 

Stage  4:  Core  process  re-engineering 
attempts.  After  a  careful  analysis  of  the 
process  of  acquisition  of  construction  ser¬ 
vices,  the  PR  group  developed  an  initial 
plan  to  radically  redesign  the  process. 
Most  of  the  analysis  was  based  in  two 


main  large  process  models:  a  workflow 
model  (Soles,  1994;  Tagg,  1996)  repre¬ 
senting  the  process  as  a  set  of  over  100 
interrelated  activities;  and  a  role-activity 
diagram  (Moretti  et  al.,  1996)  showing  the 
flow  of  documents  between  organizational 
roles  as  well  as  identifying  the  “contact 
points”  in  the  process.  Figure  1  shows  a 
simplified  workflow  representation  of  a 
small  part  the  process  (the  one  related  to 
selecting  a  service  supplier).  Names  of 
activities  and  organizational  roles  per¬ 
forming  the  activities  (shown  within  pa¬ 
rentheses)  are  in  the  rectangles. 

The  plan  devised  by  the  PR  group  in¬ 
volved  the  application  of  simple  re-engi¬ 
neering  techniques,  such  as  structuring  the 
organization  around  outcomes  rather  than 
single  tasks  (Hammer,  1990;  Davenport, 
1993)  and  reducing  unnecessary  controls 
(Hammer  and  Champy,  1993;  Kock, 
1995). 

Structuring  the  organization  around 
outcomes,  not  tasks,  implies  having  one 
person  perform  a  set  of  activities  that  pro¬ 
duce  one  single  output,  rather  than  sev¬ 
eral  people  separately  performing  each 
single  activity  (Buzacott,  1996).  The  ap¬ 
plication  of  this  principle  by  the  PR  group 
has  indeed  led  to  a  considerable  concep¬ 
tual  reduction  in  the  number  of  roles,  and 
therefore  a  decrease  in  the  number  of  con¬ 
tact  points  and  a  consequent  reduction  in 
cycle  time,  in  the  process  shown  in  Fig¬ 
ure  1.  However,  these  reductions  were 
achieved  only  from  a  conceptual  perspec¬ 
tive.  In  reality,  none  of  the  roles  in  the  pro¬ 
cess  could  be  replaced  by  any  other  role, 
for  two  main  reasons:  Different  types  of 
expertise  were  required  to  perform  differ¬ 
ent  activities;  and,  most  important,  the 
whole  process  was  originally  designed 
based  on  the  federal  and  state  laws  for 
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Figure  1:  The  Process  of  Selecting  a  Supplier 


7 


Acquisition  Review  Quarterly — Winter  1998 


public  bids,  which  considerably  limited 
the  number  of  possible  changes  in  its  role 
and  workflow  structure — even  small 
changes  could  lead  the  process  to  fall  out¬ 
side  legal  parameters. 

For  example,  some  improvements 
could  conceptually  have  been  achieved  by 
pushing  the  responsibility  of  calculating 
budgets  from  budget  consultants  to  the 
Projects  Department.  This  could  reduce 

time  wasted,  as 


"Moreover,  and  most 
important,  the  law 
required  that  an 
initial  budget  be 
produced  by  an 
"independent  group 
of  recognized  ex¬ 
perts,"  hence  the 
assignment  of  the 
|ob  to  a  team  of 
expert  budget  con¬ 
sultants  (all  em¬ 
ployed  by  a  presti¬ 
gious  consulting 
firm)." 


budget  consult¬ 
ants  typically 
had  to  wait  for  a 
project  plan  to 
be  generated 
and  sent  to  them 
by  the  Projects 
Department  be¬ 
fore  they  could 
start  working  on 
a  budget  for  the 
project  (see  Fig¬ 
ure  1).  However, 
this  would  likely 


be  achieved  at  a 
cost,  probably  a  decrease  in  the  precision 
of  quotes  and  hence  the  quality  of  the  over¬ 
all  budget,  as  the  employees  in  the  Projects 
Department  were  not  prepared  to  perform 
the  complex  calculations  involved  in  gen¬ 
erating  construction  budgets.  Moreover, 
and  most  important,  the  law  required  that 
an  initial  budget  be  produced  by  an  “in¬ 
dependent  group  of  recognized  experts,” 
hence  the  assignment  of  the  job  to  a  team 
of  expert  budget  consultants  (all  employed 
by  a  prestigious  consulting  firm).  The 
same  goes  for  the  taking  and  handling  of 
appeals  on  the  decision  made  by  the  bids 
commission.  For  example,  could  these  not 
be  performed  by  the  Bids  Division  itself? 
After  a  careful  analysis,  the  answer  was 


found  to  be  negative,  as  the  law  required 
that  a  team  of  attorneys  registered  with 
the  Brazilian  Bar  Association  be  in¬ 
volved  in  this  activity. 

Conceptually,  reducing  “unnecessary 
controls”  would  certainly  reduce  the  time 
required  to  select  a  supplier.  A  number  of 
these  “unnecessary”  controls  were  embed¬ 
ded  in  the  process,  even  though  not  all  of 
them  are  explicitly  shown  in  Figure  1 .  For 
example,  after  a  decision  had  been  made 
by  the  Bid  Commission  about  the  winner 
of  the  public  bid,  it  would  have  to  first  be 
communicated  to  the  team  of  lawyers  and 
the  Public  Relations  Department.  The  Bids 
Division  would  then  draft  a  report  on  the 
bid,  including  the  decision  made  by  the 
Bid  Commission,  which  would  then  be 
thoroughly  checked  by  the  team  of  law¬ 
yers  for  full  compliance  with  the  complex 
legal  requirements  regarding  public  bids. 
Finally  the  report  would  be  checked  again 
by  the  Public  Relations  Department, 
which  would  then  provide  a  summary  of 
the  report  to  the  public  through  media  ve¬ 
hicles — large  newspapers  and  official 
government  publications. 

These  checks  were  seen  as  extremely 
important  to  guarantee  that  no  mistakes 
would  be  committed  that  could  lead  to  the 
invalidation  of  a  public  bid  on  legal 
grounds.  Also,  several  checks  were  explic¬ 
itly  prescribed  by  the  law.  For  example, 
PubliCorp  was  required  by  law  to  set  aside 
some  time  to  handle  administrative  ap¬ 
peals  through  its  team  of  lawyers,  an  ac¬ 
tivity  that  was  included  by  legislators  in 
the  public  bid  process  to  make  sure  that 
its  execution  could  be  thoroughly  checked 
and  formally  questioned  by  all  bidders — 
and  often  bids  were  questioned,  typically 
by  those  bidders  who  were  unsuccessful 
in  the  bid. 
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Over  approximately  one  year,  several 
attempts  were  made  by  the  PR  group  to 
re-engineer  the  process  of  acquisition  of 
construction  services  and  other  core  pro¬ 
cesses  at  PubliCorp.  All  of  these  attempts 
were  consistently  unsuccessful  in  that  only 
small  process  changes  were  effected,  and 
less  than  relevant  positive  bottom-line  re¬ 
sults  (such  as  possible  cost  and  cycle  time 
reductions)  were  achieved.  The  sheer  le¬ 
gal  rigidity  of  the  process  was  singled  out 
by  a  number  of  PR  group  members  as  the 
main  impediment  to  the  success  of  these 
attempts.  One  of  these  members  pointed 
out  that  “radical  change  in  public  organi¬ 
zations  such  as  [PubliCorp]  must  be  ac¬ 
companied  by  radical  changes  in  the 
law... but  changes  in  the  law  take  time  and 
a  lot  of  lobbying  to  be  achieved....” 

Stage  5:  Process  automation.  The 
problems  faced  by  the  PR  group  led  it  on 
a  path  where  eventual  destruction  was  in 
sight  (should  it  not  be  able  to  accomplish 
at  least  part  of  its  goals).  In  the  meantime, 
however,  the  members  of  this  group  had 
not  only  become  an  established  and  cohe¬ 
sive  team  at  PubliCorp,  but  also  acquired 
considerable  power  due  to  the  frequent 
interactions  with  PubliCorp’s  chief  execu¬ 
tive  officer  (CEO)  and  board  of  directors 
during  its  more  than  three  years  of  exist¬ 
ence.  In  an  auto-preservation  attempt,  the 
PR  group  gradually  moved  away  from 
process  re-engineering  to  automation — 
that  is,  the  PR  group  began  to  simply  au¬ 
tomate  processes  rather  than  trying  to  radi¬ 
cally  redesign  them.  This  move  took  place 
in  an  almost  imperceptible  way,  as  pro¬ 
cess  automation  was  presented  by  the  PR 
group  to  the  CEO  and  senior  executives 
as  process-focused  change  and  therefore 
analogous  to  process  re-engineering. 


sheer  legal 
rigidity  of  the  pro¬ 
cess  was  singled  out 
by  a  number  of  PR 
group  members  as 
the  main  impedl- 
mehf  to  the  success 
of  these  attempts/' 


The  PR  group  tactics  have  not  gone 
unquestioned,  at  least  initially,  but  by  then 
its  political  power  within  the  organization 
was  enough  to  eliminate  any  opposition. 
For  example,  some  opposition  to  the  PR 
group  was  championed  by  PubliCorp’s 
chief  information  officer  (CIO),  who  ques¬ 
tioned  the  need 
for  the  exist¬ 
ence  of  the  PR 
group  since  it 
was  not  doing 
its  job.  The  re¬ 
action  was  swift 
and  vicious, 
leading  to  the 
officer’s  quick 

dismissal  on  a  few  dubious  charges  of  neg¬ 
ligence  and  involuntary  accessory  to  com¬ 
puter  theft. 

After  this  incident,  the  PR  group  car¬ 
ried  on  automating  processes  while  the  IT 
group  provided  the  necessary  LAN  infra¬ 
structure  support.  As  a  result,  four  years 
after  the  re-engineering  attempt  was  be¬ 
gun,  few  bottom-line  process  improve¬ 
ments  have  been  achieved,  in  spite  of  the 
over  $8  million  spent  in  the  attempt  to  re¬ 
engineer  PubliCorp.  Nevertheless,  at  least 
some  in  the  media  thought  of  the  re-engi¬ 
neering  attempt  as  a  relatively  successful 
endeavor.  Among  other  distinctions,  the 
CEO  was  hailed  by  one  local  independent 
newspaper  as  the  architect  of  a  very  suc¬ 
cessful  “modernization”  of  PubliCorp 
with  “state-of-art”  IT,  and  portrayed  as  a 
role  model  for  public  sector  managers. 
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Structural  Factors  Preventing 
Radical  Change:  A  Public  Sector  View 


Arguably,  a  number  of  factors  could 
have  contributed  to  the  failure  of  the  re¬ 
engineering  attempt  at  PubliCorp.  It  could 
have  been  argued  that  the  PR  group  acted 
unethically,  letting  their  struggle  for  power 
prevent  them  from  searching  for  genuine 
radical  process  improvements  in  the 
organization’s  processes,  or  that  the  op¬ 
position  from  employees  led  to  the  fail¬ 
ure  of  the  re-engineering  attempt.  It  could 


also  have  been  argued  that  there  was  not 
enough  top  management  support  for  radi¬ 
cal  change,  as  the  CEO  apparently  chose 
to  ignore  the  warnings  of  his  CIO  about 

the  PR  group’s 


^'Process  functional 
lieterogeneity  can 
be  measured  by 
Bounting  the  number 
»f  different  organi- 
cational  functions  or 
learns  involved  in 
Ihe  execution  of  a 
process  (e.g.,  CEO, 
budget  consultants, 
lawyers  team)." 


ineffectiveness, 
and  accept  the 
use  of  the  pro¬ 
ject  by  the  local 
media  as  an  ex¬ 
ample  of  a  suc¬ 
cessful  attempt 
to  modernize  a 
public  organiza¬ 
tion.  Finally,  it 
could  have  been 


argued  that  there 
was  not  a  clear  understanding  from  the 


part  of  the  CEO  or  the  PR  group  about 
what  radical  process  improvement  entails, 
and  how  it  should  be  properly  conducted. 

However,  should  all  the  above  circum¬ 
stances  be  modified  so  as  to  favor  re-en¬ 
gineering,  there  would  still  be  a  major 
obstacle  to  be  overcome  at  PubliCorp — 


the  process  rigidity  imposed  by  the  Bra¬ 
zilian  government  regulation.  Other  than 
employees’  resistance  to  change,  the  case 
study  suggests  the  existence  of  a  structural 
resistance  to  change  built  in  the  organiza¬ 


tional  process.  Structural  resistance  in  the 
core  acquisition  process  targeted  for  re¬ 
engineering  at  PubliCorp  can  be  seen  as 
the  resistance  embedded  in  the  process  it¬ 
self,  not  only  due  to  the  way  activities  were 
designed  to  be  carried  out  and  by  whom, 
but  mainly  due  to  the  fact  that  this  design 
was  set  out  in  the  form  of  governmental 
legislation.  PubliCorp’s  case  suggests  an 
avenue  for  the  understanding  of  process 
rigidity  in  the  public  sector,  whereby  it  can 
be  viewed  as  a  function  of  at  least  two 
process  variables — functional  heterogene¬ 
ity  and  degree  of  regulation.  Table  1  is  an 
attempt  to  summarize  this  understanding 
into  a  two-by-two  matrix. 

Process  functional  heterogeneity  can  be 
measured  by  counting  the  number  of  dif¬ 
ferent  organizational  functions  or  teams 
involved  in  the  execution  of  a  process 
(e.g.,  CEO,  budget  consultants,  lawyers 
team).  Functional  heterogeneity  in  orga¬ 
nizational  processes  has  been  found  to  be 
highly  correlated  with  the  number  of 
knowledge  specialization  areas  found  in 
processes  (Kock  and  McQueen,  1996) — 
that  is,  the  number  of  different  types  of 
expertise  required  to  perform  a  process. 
Due  to  virtually  insurmountable  obstacles 
to  a  person  becoming  an  expert  in  several 
areas  at  the  same  time,  career  choices  are 
made  that  lead  to  knowledge  specializa¬ 
tion.  A  related  consequence  is  that  func¬ 
tional  heterogeneity  is  likely  to  be  high  in 
many  processes  carried  out  within  knowl¬ 
edge-intensive  organizations  (Kock  et  al., 
1996). 

Table  1  concisely  states  that  a  high 
functional  heterogeneity  combines  with  a 
high  degree  of  process  regulation  to  gen¬ 
erate  a  high  structural  rigidity  in  organi¬ 
zational  processes.  The  case  study  sup¬ 
ports  this  conjecture  and  provides  the 
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Table  1: 

Structural  Rigidity  as  a  Function  of  Functionai  Heterogeneity 
and  Degree  of  Regulation 


Medium  rigidity 

High  rigidity 

High  functional  heterogeneity 

For  example,  semi-autonomous 
public  and  knowledge-intensive 
institutions  such  as  state 
universities. 

For  example,  public  and 
knowledge-intensive 
companies  such  as  inspection 
firms  in  knowledge-intensive 
industries  (PublicCorp). 

(Re-engineering  is  likely  to  fail 
without  changes  in  legislation.) 

(Re-engineering  is  very  likely 
to  fail  without  changes  in 
legislation.) 

Low  rigidity 

Medium  rigidity 

Low  functional  heterogeneity 

For  example,  semi-autonomous 
government  Inspection 
branches  in  non  knowledge- 
intensive  areas  such  as  farm 
inspection  departments. 

For  example,  government 
inspection  branches  In  non 
knowledge-intensive  areas 
such  as  internal  revenue 
services. 

(Re-engineering  may  succeed 
without  changes  in  legislation.) 

(Re-engineering  is  likely  to  fail 
without  changes  in  legislation.) 

Low  degree  of  regulation 

High  degree  of  regulation 

basis  for  the  understanding  of  the  dynam¬ 
ics  through  which  process  rigidity  opposes 
attempts  to  radical  process  redesign.  De¬ 
mands  for  high  functional  heterogeneity 
lead  to  a  high  number  of  functional  roles 
in  processes  (e.g.,  budget  consultants  and 
lawyers  in  PubliCorp’s  acquisition  pro¬ 
cess),  which  are  then  the  focal  point 
around  which  government  regulation  is 
created  and  passed — for  example,  the  le¬ 
gal  prescription  that  construction  budgets 
be  prepared  by  a  group  of  “recognized 
experts,”  the  budget  consultants,  and  the 
related  criteria  prescribed  in  law  to  iden¬ 
tify  and  hire  these  experts. 

Government  regulation  solidifies  the 
procedures  involving  each  one  of  the  or¬ 
ganizational  functions  performing  process 
activities,  turning  each  function  into  a 
potential  focus  of  resistance  against 


radical  change.  PubliCorp’s  case  shows 
that  the  line  people  who  carry  out  knowl¬ 
edge-intensive  activities  in  government- 
regulated  processes  are  likely  to  be  more 
familiar  with  the  change  constraints  im¬ 
posed  by  government  regulation  on  those 
activities  than  consultants  and  managers. 
After  all,  line  workers  know  their  work 
better  than  others  not  directly  involved  in 
it  (Deming,  1986).  Hence,  it  often  be¬ 
comes  their  duty  to  repeatedly  inform  the 
members  of  re-engineering  teams  (e.g.,  the 
PR  group  at  PubliCorp)  that  radical  rede¬ 
sign  cannot  be  achieved  the  way  it  is  pro¬ 
posed.  This  may  lead  to  communication 
breakdowns  between  line  workers  and  re¬ 
engineers  as  the  former  group  sees  the  lat¬ 
ter  group  as  ineffective,  and  the  latter 
group  sees  the  former  as  a  biased  source 
of  information  that  does  not  seem  to  favor 
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the  re-engineering  attempt  anyway.  The 
final  result  is  likely  to  be  a  failed  re-engi¬ 
neering  attempt. 


What  About  the  Success  Stories? 


Some  of  our  conclusions  may  be  ques¬ 
tioned  based  on  successful  examples  of 
process  change  in  the  public  sector.  But 
an  analysis  of  successful  cases  reported 

so  far  does  not 


^^here  are  very  few 
published  public 
sector  cases  of  suc¬ 
cessful  re-engineer¬ 
ing  in  situations  of 
high  rigidity,  rela¬ 
tive  to  the  number  of 
cases  about  re-engi¬ 
neering  in  low  rigid¬ 
ity  (usually  fully 
private)  organiza¬ 
tions/' 


invalidate  our 
conclusions,  but 
provides  the 
background  on 
which  to  frame 
our  understand¬ 
ing  of  process- 
related  change 
in  the  public 
sector  and  what 
it  entails.  There 


have  been  ex¬ 
amples  of  successful  outsourcing  of  core 
public  services  or  large  components  of 
these  services  to  private  companies 
(Coppell,  1994;  Mukherjee  and  Braganza, 
1994;  Williams,  1994);  and  of  the  suc¬ 
cessful  transfer  of  modified  government 
functions  to  the  community  served  by 
local  government  departments  (e.g., 
police  departments)  through  what  are 
often  referred  to  as  “community  empow¬ 
erment”  initiatives  (Osborne  and  Gaebler, 
1992,  Chapter  2). 

Some  of  the  examples  above  can  be 
seen  as  instances  of  successful  re-engi¬ 
neering  projects  by  some,  as  they  might 
indeed  have  led  to  radical  change  in  core 
governmental  processes.  However,  they 
do  not  satisfy  one  basic  criterion  to  be 
considered  in  re-engineering  projects — 


that  at  least  one  organizational  process  be 
radically  redesigned,  leading  to  a  radical 
improvement  in  outcome  quality  or  pro¬ 
ductivity  of  the  process  while  it  (the  pro¬ 
cess)  is  still  part  of  the  organization.  The 
above  examples  of  successful  change  in 
the  public  sector  have  relied  heavily  on 
the  transfer  of  whole  processes  or  parts  of 
these  of  these  processes  to  private  hands 
or  to  the  community.  After  the  processes 
had  been  farmed  out,  the  stage  was  set  for 
radical  process  change.  In  these  cases, 
however,  radical  change  occurs  when  the 
processes  are  outside  the  public  organ¬ 
ization’s  boundaries.  From  this  perspec¬ 
tive,  these  initiatives  resemble  much  more 
privatization  than  re-engineering,  as  the 
processes  are  no  longer  part  of  the  pub¬ 
lic  organization  when  they  are  radically 
redesigned. 

There  are  very  few  published  public 
sector  cases  of  successful  re-engineering 
in  situations  of  high  rigidity,  relative  to 
the  number  of  cases  about  re-engineering 
in  low  rigidity  (usually  fully  private)  or¬ 
ganizations  (although  there  have  been  re¬ 
ports  of  successful  “nontraditional”  re¬ 
engineering  in  high-rigidity  organizations, 
such  as  “knowledge-based  re-engineer¬ 
ing,”  where  part  of  the  expert  knowledge 
involved  in  carrying  out  process  activities 
is  built  in  knowledge-based  systems;  see 
Nissen,  1997). 

Even  in  situations  of  medium  rigidity, 
apparently  successful  cases  of  radical 
change  in  public  institutions  and  compa¬ 
nies  or  government  departments  resemble 
more  massive  downsizing,  where  the  fo¬ 
cus  is  on  reducing  the  size  of  the  organi¬ 
zation  by  shedding  off  apparently  unnec¬ 
essary  departments  and  personnel,  than  re¬ 
engineering,  whose  focus  is  on  radically 
redesigning  organizational  processes. 
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Many  such  examples  exist,  as  the  case  of 
the  New  Zealand  Ministry  of  Agriculture 
and  Fisheries,  which  downsized  thirteen 
divisions  with  6,000  employees  in  the  late 
1980s  into  five  divisions  with  2,600  em¬ 
ployees  in  the  early  1990s  (French,  1994), 
and  the  case  of  13  Swedish  government 
agencies  that  laid  off  half  of  their  employ¬ 
ees  in  one  blow  in  1990  (Naschold  and 
Otter,  1996). 

COHCmSIOM  AND  IMPLICATIONS _ 

We  have  discussed  in  this  paper  the  role 
of  structural  process  rigidity  as  an  obstacle 
to  radical  process-based  change  in  public 
sector  organizations.  A  preliminary  frame¬ 
work  for  understanding  process  rigidity  is 
proposed,  where  two  main  influences  are 
identified.  The  framework  proposes  that 
process  rigidity  is  influenced  by  the  func¬ 
tional  heterogeneity  of  processes,  and  the 
degree  of  government  regulation  imposed 
on  processes  in  public  sector  organiza¬ 
tions.  We  argue  that  high-rigidity  pro¬ 
cesses  may  not  be  amenable  to  radical  re¬ 
design  if  they  are  not  outsourced  to  less 
regulated  organizations  such  as  private 
companies  and  community  associations. 
We  base  our  argumentation  on  the  analy¬ 
sis  of  a  failed  attempt  to  re-engineer  a  ser¬ 
vice  acquisition  process  in  a  public  sector 
organization,  and  on  some  cases  from  the 
literature  on  public  sector  transformation. 


Two  main  implications  for  public  sec¬ 
tor  organizations  stem  from  this  research. 
First,  public  sector  organizations  should 
question  whether  radical  change  is  really 
necessary  before  embarking  on  large-scale 
and  radical  process  improvement  at¬ 
tempts,  as  the  probability  of  failure  in 
these  attempts  is  necessarily  high.  Incre¬ 
mental  improvement  initiatives  have 
proven  to  be  less  risky,  while  having  in 
many  cases  yielded  highly  satisfactory 
organizational  improvements,  particularly 
in  process  outcome  quality  (Koehler  and 
Pankowski,  1996;  Raff  and  Beedon, 
1994). 

Second,  public  sector  organizations 
where  radical  change  is  seen  as  absolutely 
necessary  can  benefit  from  the  analysis  of 
their  processes  regarding  their  functional 
heterogeneity  and  degree  of  regulation 
before  initiating  their  radical  change 
projects.  Whenever  radical  changes  in  the 
law  are  possible  in  a  short  period  of  time 
(i.e.,  less  than  one  year),  high-rigidity  or¬ 
ganizations  may  consider  embarking  on 
process  re-engineering  attempts.  How¬ 
ever,  since  radical  changes  in  law  usu¬ 
ally  cannot  be  quickly  accomplished  in 
most  democratic  countries,  most  high-ri- 
gidity  organizations  in  the  public  sectors 
within  these  countries  are  likely  to  be 
more  successful  if  they  move  towards  other 
alternative  approaches  to  radical  change 
such  as  privatization  and  community 
empowerment. 
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TUTORIAL 


A  MULTIMEDIA  SYSTEMS  APPROACH 
TO  NATIONAL  SECURITY  POLia, 
DECISION  MAKING,  AND 
INTELLIGENCE  SUPPORT 


Sohn  R  Crecme  and  Michael  D,  Salomone 


The  amount  of  raw  data  available  to  military  and  national  Intelligence 
professionals  is  growing  exponentially.  The  revolution  in  our  ability  to  generate 
and  disseminate  data  has  not  led  to  a  corresponding  increase  in  our  ability  to 
convert  data  to  useful  information.  The  revolution  in  information  technology, 
coupled  with  the  natural  tendencies  in  military  and  intelligence  organizations 
to  compartmentalize  data  (especially  when  collected  using  automated  or  covert 
methods),  combine  to  hinder  military  commanders,  top-level  analysts,  and 
national  decision  makers  from  coherently  Interpreting  this  profusion  of 
information.  The  “stove-piping”  of  data  within  subunits  of  an  organization  makes 
effective  integration  at  the  top  much  more  difficult. 

Existing  data  processing  systems  make  some  kinds  of  information  extremely 
useful,  but  the  systems  are  incapable  of  storing,  organizing,  processing,  and 
retrieving  other  data  types,  including  text  and  images,  in  ways  that  allow  top- 
level  decision  makers  and  analysts  to  make  coherent,  comprehensive  use  of 
the  information.  Systems  based  on  new  technologies  can  enable  these 
professionals  to  more  efficiently  manage  data  and  information  available  in 
electronic  forms.  The  combination  of  an  object-relational  database 
management  system  with  a  natural-language  processing-based,  free-form  text 
search  engine,  paired  with  image  recognition  software  and  standard  structured 
query  language-based  search,  combined  with  recent  advances  in  processing, 
storage,  display,  and  telecommunications  technologies,  holds  great  promise 
for  effective  information  management  systems  of  the  future. 


Intelligence  is  information.  Raw  data 
from  multiple  sources  and  in  a  vari¬ 
ety  of  forms,  brought  together  in  a 


coherent  and  timely  way,  integrated  and 
congealed  in  the  context  of  particular 
policy  and  operational  issues  or  in  the 
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interest  of  a  comprehensive  interpretation 
of  particular  events,  is  transformed  into 
useful  information.  The  challenge  of  man¬ 
aging  data  available  in  digital  and  elec¬ 
tronic  form,  as  it  is  transformed  into  use¬ 
ful  information,  is  enormous — both  in 
terms  of  the  vast  quantities  of  data  and  in 
terms  of  the  incredible  diversity  and 
complexity  of  the  raw  data  elements 
routinely  collected.  The  fundamental 
problem  is  that  a  huge,  eclectic,  and  rich 
array  of  data  must  be  organized  and  stored 
in  such  a  way  as  to  allow  for  retrieval  in 
targeted  and  often  unanticipated  ways  for 
unanticipated  purposes. 

The  net  effect  of  modem  information 
and  communications  technologies  has 
been  to  exacerbate  rather  than  alleviate  the 
information  management  problem  for  all 
professionals,  not  just  those  in  the  mili¬ 
tary  and  intelligence  communities.  Com- 
partmentalization  and  division  of  labor, 


the  hallmark  of  organization  and  bureau¬ 
cracy,  creates  dedicated,  specialized,  and 
separate  information  management  and 
data  processing  systems.  This  “stove-pip¬ 
ing”  effect  means  that  different  data  types 
and  data  collection  organizations  have  a 
tendency  to  operate  as  closed  systems,  up 
through  several  organizational  levels. 
Once  data  has  been  processed  through 
several  organizational  levels,  the  initial 
data  elements  or  observations  or  transac¬ 
tions  are  “lost”  or  unavailable  to  top-level 
analysts  or  decision  makers.  Whether  the 
initial  data  element  is  a  paragraph  in  a 
trade  journal  or  an  internal  government 
memo,  or  is  a  portion  of  the  yield  from  a 
satellite’s  remote  sensor,  a  readiness  re¬ 
port  for  a  low-level  military  unit,  or  a 
simple  accounting  transaction,  the  raw 
data  is  seldom  easily  accessible  to  top- 
level  analysts  or  decision  makers,  and  it 
is  almost  always  impossible  to  combine 
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data  from  different  sources  (“stovepipes”) 
to  suit  unanticipated  needs.  Modem  infor¬ 
mation  technology  generally  accentuates 
and  multiplies  the  stove-piping  problem. 

As  we  move  into  the  next  century  with 
an  accelerating  pace  of  technological 
change,  an  important  question  is  whether 
modem  information  and  communications 
technologies  can  be  used  to  alleviate  a 
problem  they  have  helped  create.  Can 
technology  aid  in  the  targeted  retrieval  and 
purposeful  synthesis  of  the  technology- 
driven  explosion  in  the  quantity  and  di¬ 
versity  of  raw  data  in  analog  and  digital 
formats?  Just  as  it  took  an  important  en¬ 
abling  technology — HTML  (HyperText 
Markup  Language),  Web  browsers  (Mo¬ 
saic/Netscape),  and  search  engines  (Lycos, 
Excite,  AltaVista,  Yahoo) — to  make  the 
Internet  useful  for  tens  of  millions  of  us¬ 
ers  with  eclectic  interests,  military,  intel¬ 
ligence  and  other  professionals  need  en¬ 
abling  technologies  to  make  the  rapidly 
growing  mountains  of  increasingly  diverse 
data  types  professionally  purposeful. 

In  the  language  of  the  national  security 
and  intelligence  community,  “Can  mod¬ 
em  information  technology  rescue  the  in¬ 
telligence  fusion  process  from  the  enor¬ 
mous  information  overload,  storage,  re¬ 
trieval,  and  synthesis  problem?” 

The  Nature  of  the  Data  ahd 

IHFORMATIOH  MAHACEMEHT  PROBLEM 

This  challenge  is  complicated,  since 
data  relevant  to  a  particular  issue  is  col¬ 
lected  from  a  very  wide  array  of  sources, 
usually  by  different  people  in  different  or¬ 
ganizations,  and  often  for  narrower  pur¬ 
poses  than  those  of  relevance  to  national 
intelligence  organizations,  policy  makers. 


or  a  military  commander.  The  efficient  and 
cost-effective  management  of  potentially 
relevant  data  and  information  is  a  central 
issue  for  all  organizations,  and  particularly 
so  for  the  top-level  decision  making, 
policy,  and  intelligence  functions  in  mili¬ 
tary  organizations  and  the  national  intel¬ 
ligence  commu- 

nity.  Data  and  effleient  and 

information  rel-  ,ost.eflee#ive  man. 
evant  to  these  agement  of  poten- 
functions  is  in-  tially  relevant  data 
credibly  di-  and  information  U  a 
verse,  compiled  central  Issue  for  all 
in  great  quanti-  organixations..." 
ties,  in  widely 

varying  formats,  and  from  wide-ranging 
and  shifting  numbers  of  sources. 

Much  of  the  data  waiting  to  be  collated 
and  synthesized  into  information  are 
memos,  journal  articles,  news  accounts 
from  the  print  media,  reprints,  translations, 
notes,  papers,  and  transcripts — free-form 
text.  Still  other  data  take  the  form  of  photo 
images,  sensor  outputs,  signals,  video 
images,  tabular  alpha-numeric  data,  maps, 
news  accounts  from  the  electronic  media, 
or  audio  clips.  Data  and  information  re¬ 
sources  are  generally  dispersed  geographi¬ 
cally  and  organizationally.  The  manage¬ 
ment  of  information  for  top-level  decision 
making,  policy,  and  intelligence  functions 
is  a  formidable  task  and,  as  currently  con¬ 
ducted,  is  necessarily  very  labor  intensive. 

Consider  only  the  accelerating  informa¬ 
tion  management  sub-task  of  battlefield 
management.  As  new  sensors  and  intelli¬ 
gence  gathering  devices  are  created  to 
support  development  of  dominant  battle 
space  awareness,  there  is  a  corresponding 
and  geometric  increase  in  the  amount  of 
information  the  commander  and  the  sup¬ 
porting  intelligence  organizations  must 


21 


Acquisition  Review  Qtfcrrfer/y— -IV/nfer  1998 


comprehend,  at  least  in  theory.  Effective 
commanders  must  be  increasingly  selec¬ 
tive  in  the  information  to  which  they  at¬ 


tend,  or  they  need  to  capture  the  benefits 
of  automation  in  executing  the  informa¬ 
tion  management  portion  of  their  com¬ 
mand  function. 

Consider  the  real-time  intelligence  ana¬ 
lysts’  task  of  in¬ 
terpreting  rap¬ 
idly  changing 
events,  with  a 
pile  of  articles, 
notes,  demo¬ 
graphic  charts, 
biographical 
sheets,  maps, 
books.  Cable 
News  Network 
(CNN)  video 
tapes,  and 
newspaper  clippings  on  his  or  her  desk. 
First,  is  this  eclectic  pile  of  data  the  right 
pile  for  this  event?  Second,  can  the  ana¬ 
lyst  quickly  sort  through,  analyze,  cross- 
reference,  and  correlate  information  in 
support  of  a  competent  situational  analy¬ 
sis?  Under  the  best  of  circumstances,  this 


"In  a  bureaucratic 
setting— be  it  a 
military  or  intelli¬ 
gence  organization- 
information  seldom 
is  delivered  to  a 
commander  or  ana¬ 
lyst  without  an 
associated  context, 
framework,  or  inter¬ 
pretive  suggestion. 


## 


a  daunting  information  storage,  retrieval, 
and  organizational  task. 

The  problems  faced  by  the  commander 
in  a  battlefield  situation  or  by  an  intelli¬ 
gence  analyst  are  surprisingly  similar.  Due 
to  the  pressures  of  time  or  the  overwhelm¬ 
ing  amounts  of  potentially  relevant  infor¬ 
mation,  there  is  an  information  overload 
problem  of  considerable  proportions. 
Whether  individuals  are  seeking  the  “right 
answer,”  an  accurate  situational  analysis, 
an  adequate  understanding  of  events,  or  a 


framework  for  understanding  the  phenom¬ 
ena  or  event  of  interest,  individuals  must 
be  selective,  focused,  and  guided  by  some 
sort  of  paradigm  or  crude  framework  in 
interpreting  the  data  they  have.  For  the 
most  part,  history  and  experience,  shaped 
by  training,  dominant  scenarios,  and  “les¬ 
sons  learned”  from  roughly  similar  situa¬ 
tions,  help  individuals  sort  through  the  in¬ 
finite  combinations  and  permutations  of 
potentially  relevant  information.^ 

In  a  bureaucratic  setting — be  it  a  mili¬ 
tary  or  intelligence  organization — infor¬ 
mation  seldom  is  delivered  to  a  com¬ 
mander  or  analyst  without  an  associated 
context,  framework,  or  interpretive  sug¬ 
gestion.  Often  organizational  subunits  are 
formed  around  a  particular  paradigm  or 
belief  system  and  information  coming 
from  such  organizational  units  comes 
laden  with  those  biases  and  selectivity  fil¬ 
ters.  Organizational  preprocessing  is 
hardly  unexpected.  The  various  elements 
of  the  technology-driven  information 
revolution  combine  to  deliver  vastly 
greater  quantities  of  information  in  vastly 
more  diverse  formats  (data  types),  result¬ 
ing  in  a  geometric  growth  in  the  potential 
information  processing  and  synthesis 
problem.  This  problem  is  made  manage¬ 
able  in  current  settings  by  even  greater 
reliance  on  dominant  paradigms  generated 
by  personal  histories  and  training  and  by 
even  greater  selectivity  and  filtering  by 
bureaucratic  information  providers.  The 
information  revolution  has  been,  in  fact, 
a  revolution  in  data  generation  and  dis^ 
semination,  not  in  data  understanding. 

The  information  revolution  does  not 
naturally  create  a  greater  ability  of  chief 


^  This  is  illustrated  in  Graham  T.  Allison’s  classic  Essence  of  Decision,  (Little  Brown,  1969)  in  analyzing  the 
Cuban  Missile  Crisis  using  three  theoretical  approaches  or  points  of  view. 
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executive  officers,  commanders,  and  ana¬ 
lysts  to  “get  it  right.”  Here  we’ll  discuss 
some  of  the  technical  reasons  why  the 
revolution  in  our  ability  to  generate  and 
disseminate  data  has  not  led  to  a  corre¬ 
sponding  increase  in  our  ability  to  con¬ 
vert  data  to  useful  information.  We’ll  de¬ 
scribe  some  technical  developments  that 
we  believe  can  help  improve  the  data-un- 
derstanding-to-useful-information  conver¬ 
sion  process,  through  a  partial  automation 
of  the  new,  more  diverse,  and  difficult  in¬ 
formation  management  task. 

The  explosion  in  the  volume  and  diver¬ 
sity  of  intelligence  information  is  occur¬ 
ring  in  an  environment  of  declining  finan¬ 
cial  resources  for  intelligence  assessment, 
military  decision  making,  and  national 
security  policy  making.  Professionals  in 
these  areas  must  achieve  dramatic  in¬ 
creases  in  the  efficiency  and  effectiveness 
of  their  information  management  and  fu¬ 
sion  systems,  merely  to  maintain  current 
levels  of  effectiveness  of  intelligence,  de¬ 
cision  making,  and  policy  functions. 


Applying  Recent  Advances  in 
Technology  to  the  Challenge 


How  might  one  bring  modem  informa- 
tion  processing  technology  to  bear  on  na¬ 
tional  intelligence  tasks?  Recent  advances 
in  many  areas  foster  the  hope  of  creating 
affordable,  comprehensive  information 
systems.  Among  these  are: 

•  multimedia-data  applications, 

•  analog-to-digital  conversions  and  data 
compression, 


•  artificial  intelligence  applications  in 
image  recognition  and  natural  language 
processing, 

•  modem  telecommunications,  including 
both  wide-area  and  local-area  network 
technologies, 

•  client/server  computing  systems, 

•  distributed  database  systems, 

•  visualization  and  display  technologies, 

•  digital  storage  media,  and 

•  computing  technologies. 


Systems  based  on  new  technologies  can 
enable  government  and  military  person¬ 
nel  to  more  efficiently  manage  the  expo¬ 
nential  growth  rate  of  data  and  informa¬ 
tion  available  in  electronic  forms,  and 


to  more  effec¬ 
tively  deal  with 
the  challenges 
presented  by 
the  prolifera¬ 
tion  of  data  for¬ 
mats.  A  com¬ 
prehensive  and 
affordable  ap¬ 
proach  to  the  in¬ 
telligence  infor¬ 
mation  man- 


"Major  advances 
hove  occurred 
recently  in  the 
ability  to  store, 
access,  and  process 
vast  amounts  of  rcm 
data  in  a  variety  of 
forms,  and  to  do  so 
using  digital  data 
formats*" 


agement  task  is  now  possible  due  to  ad¬ 
vances  on  several  technical  fronts. 

Major  advances  have  occurred  recently 
in  the  ability  to  store,  access,  and  pro¬ 
cess  vast  amounts  of  raw  data  in  a  vari¬ 
ety  of  forms,  and  to  do  so  using  digital 
data  formats.  First,  recent  advances  in 
optical  and  magnetic  technologies  make 
the  storage  and  retrieval  of  terabytes  (a 
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million  megabytes)  of  digital  information 
possible  and  affordable,  A  “jukebox”  of 
high-capacity,  read-write  compact  disks 
now  costs  less  than  $5,000  and  stores  a 
terabyte  of  information.  These  new  juke¬ 
boxes  can  replace  a  $2  million  to  $3  mil- 

«  ..II  is  the  advance 
in  MormatUni  Btor.  art  only  one  or 
age  and  retrieval  two  years  ago. 
strategies  (database  And  the  new, 
management  sys-  digital  video 
terns,  or  DBMSs)  that  disks  (with  an 
makes  It  possible  order  of  magni- 
fer  professionals  to  greater  ca- 

facilitale  the  appro-  a^g 

priate  c<dlection  of  ^ 

pandinVset  of  exile-  °tder  to  grasp 
ientlally  growing,  the  magniu^e 
data  haystacks."  I  of  this  available 

ity,  a  terabyte  represents  more  text  than  is 


stored  in  most  university  libraries. 

Second,  rapid  advances  in  video  com¬ 
pression  technologies,  driven  by  the 
broadcast  entertainment  industry,  have  re¬ 
sulted  in  the  electronic  storage  and  re¬ 
trieval  of  complex,  high-resolution  digi¬ 
tal  images,  full-motion  high  definition 
televison  (HDTV),  and  animations  and 
simulations.  Data  compression  and  expan¬ 
sion  involves  conversion  to  a  digital  for¬ 
mat,  which  increases  the  accuracy  and  re¬ 
liability  of  communication  transmissions 
and  provides  for  a  greater  ability  to  ma¬ 
nipulate  sensor  and  image  data. 

Third,  the  super  computers  of  a  couple 
of  years  ago  are  now  the  reduced  in¬ 
struction  set  computing  (RISC)  chips 
in  personal,  desktop  computers,  at  2-5 
percent  of  the  cost.  Processing  power  for 
digital  data  formats  is  available,  increas¬ 
ingly  powerful,  and  inexpensive.  Parallel 


processing  PCs  are  now  readily  available 
for  $5,000  to  $10,000  (usually  configured 
for  image  processing  and  multimedia  pro¬ 
duction  tasks). 

Finally,  the  information  superhighway, 
fiber  optic  cabling  systems,  digital  direct 
broadcasts,  cable,  and  satellite  systems 
coupled  with  advances  in  client/server  ar¬ 
chitectures  all  mean  that  the  collection  and 
storage  of  information  can  be  geographi¬ 
cally  and  organizationally  distributed, 
without  paying  the  usual  penalties  (e.g., 
lack  of  access  or  incompatibilities). 

The  rate  of  change  in  technology  is  so 
rapid,  the  efficiencies  to  be  gained  so 
great,  and  the  cost  savings  so  massive,  that 
to  use  anything  other  than  commercial, 
off-the-shelf  (COTS)  hardware  and  soft¬ 
ware  would  be  foolhardy.  An  acquisition 
process  that  fails  to  closely  track  the  mas¬ 
sive,  rapid,  and  revolutionary  advances  of 
the  commercial  sectors  of  modem  infor¬ 
mation  technology  is  fatally  flawed. 

For  these  advances,  largely  in  hardware 
and  microelectronics,  to  lead  to  real  ad¬ 
vances  in  the  information  management 
side  of  military  and  intelligence  functions, 
there  must  be  corresponding  advances  on 
the  information  processing  or  software 
portions  of  the  information  management 
system. 


Enabling  Technologies:  Database 
Management  and  the  Problems  of 
Free-Form  Text  and  Images 


Although  advances  in  hardware,  stor¬ 
age,  networking,  and  display  technologies 
make  many  storage  and  retrieval  tasks 
economically  and  technically  feasible,  it 
is  the  advance  in  information  storage  and 
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retrieval  strategies  (database  management 
systems,  or  DBMSs)  that  makes  it  pos¬ 
sible  for  professionals  to  facilitate  the  ap¬ 
propriate  collection  of  needles  in  an  ex¬ 
panding  set  of  exponentially  growing,  data 
haystacks.  Data  are  generally  in  locations 
remote  from  users.  Different  databases 
assembled,  owned,  and  maintained  by  dif¬ 
ferent  agencies  and  sources  can  be  distrib¬ 
uted  over  a  secure  network.  Distributed 
databases  can  be  comprehensively 
searched  and  information  in  multiple 
forms  can  be  selectively  retrieved  and 
brought  to  bear  on  questions  of  particular 
interest  to  analysts  or  policy  makers  in  a 
timely  fashion. 

Requirements  for  Access  to  Useful 
Information:  Database  System  and 
Search  Engine 


Two  new  areas  of  progress  combine  to 
make  possible  a  broad,  general  purpose 
national  security  and  intelligence  informa¬ 
tion  management  system.  One  area  con¬ 
sists  of  a  new  approach  to  database  man¬ 
agement:  a  data  storage  and  organization 
strategy.  For  example,  existing  relational 
database  management  systems  (RDBMSs) 
allow  the  user  to  identify  books  in  a  data¬ 
base,  based  on  a  Library  of  Congress  code 
or  index.  A  new,  object-oriented  database 
management  system  (ODBMS)  allows  a 
user  to  examine  not  only  the  Library  of 
Congress  code,  but  all  of  the  text  in  all 
the  books  in  the  database  and  go  straight 
to  the  relevant  passage  or  paragraph. 

The  second  area  relates  to  “search  en¬ 
gines”  for  free-form  text  and  images,  a 
data  search  and  retrieval  strategy.  This  con¬ 
sists  of,  first,  a  natural  language  processing 


“search  engine,”  where  concepts  and  ab¬ 
stract  representations  of  content,  as  well 
as  key  words,  are  used  to  search  and  iden¬ 
tify  relevant  information  (the  needles) 
among  the  wide  array  of  data  types  (the 
haystacks),  and 
in  afar  more 

m™  r.<u<,  r.l<iH.md 
than  has  ever  j^abases  have 

been  possible,  formed  the  basis 
Second,  it  in-  for  most  of  fhe 
eludes  image  aulomafioii  of 
recognition  adminisf ratiye  and 
software  that  al-  financial  informa« 
lows  one  tol 
search  digital 

images  (still  or  video)  based  on  color, 
composition,  objects,  structure,  or  other 
characteristics  to  identify  particular 
records  that  correspond  to  search  criteria 
or  user-provided  examples. 

Reutional  Database  Management 
Systems  and  their  Limitations _ 

For  more  than  25  years,  relational  da¬ 
tabases  have  formed  the  basis  for  most  of 
the  automation  of  administrative  and  fi¬ 
nancial  information  systems.  Relational 
databases  allow  multiple  users  to  system¬ 
atically  search  a  range  of  different,  fixed- 
record-length  files.  Fixed  record  length 
means  relational  databases  work  very  well 
for  numbers  and  alphanumeric  data  of 
fixed  length  (e.g.,  a  35-character  record 
containing  formal  names).  In  the  search 
and  retrieval  process,  the  RDBMS  can  ex¬ 
amine  individual  records  using  a  standard 
query  format  (SQL,  or  stmetured  query  lan¬ 
guage,  the  industry  standard),  and  pick  out 
those  records  that  meet  the  criteria. 
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When  data  is  located  in  complex  mul¬ 
timedia  formats  (e,g.,  a  journal  or  news 
article  or  free-form  text,  a  video,  an  im¬ 
age,  a  computer-aided  design  [CAD] 
drawing,  or  an  audio  recording),  tradi¬ 
tional  relational  databases  store  data  ele¬ 
ments  as  uninterpretable  BLOBs  (Binary 
Large  Objects).  Customarily,  relational  da¬ 
tabase  users  will  develop  a  coding  scheme 
or  “tag”  for  each  type  of  BLOB.  The  data 
or  content  in  any  particular  type  of  BLOB 
is  accessible  only  through  the  coding 
scheme.  For  example,  the  text  and  con¬ 
tent  of  a  library  book  is  stored  as  a  BLOB 
in  a  RDBMS,  and  the  BLOB  is  accessed 
through  the  Library  of  Congress  code  or 
index.  The  establishment  of  a  fixed  cod¬ 


ing  scheme,  in  effect,  implies  that  all  fu¬ 
ture  uses  for  the  content  of  a  BLOB  data 

type  are  known 

"Moreover,  and  most 
important,  the  law 
required  that  an 
initiai  budget  be 
produced  by  an 
"independent  group 
of  recognized  ex¬ 
perts,"  hence  the 
assignment  of  the 
job  to  a  team  of 
expert  budget  con- 
suitants  (ali  em¬ 
ployed  by  a  presti¬ 
gious  consulting 
firm)." 


and  have  been 
embedded  in 
the  code.  In  the 
book-in-a-li- 
brary  example, 
the  implication 
is  that  the  Li¬ 
brary  of  Con¬ 
gress  code  for 
the  book  cap¬ 
tures  all  of  the 
relevant  content 
of  the  book. 
This  assump¬ 
tion  is  seldom 


appropriate  for  the  needs  of  top-level  ana¬ 
lysts,  decision  makers,  and  policy  mak¬ 
ers,  and  is  almost  certainly  inappropriate 
for  any  knowledge  domain  that  is  rapidly 
changing.  Coding  schemes  in  a  RDBMS 
are  difficult  to  extend,  once  established. 

In  response  to  the  development  of  ob¬ 
ject-oriented  programming  and  database 


technologies,  most  leading  RDBMS  sup¬ 
pliers  have  added  what  they  term  as  an 
“object  layer”  to  their  RDBMS.  The  ob¬ 
ject  layer  generally  refers  to  a  coding 
scheme  used  to  provide  access  to  data 
stored  as  BLOBs. 


Object-oriented  Database 
Management  Systems:  Overcoming 
RDBMS  Limitations 


In  terms  of  data  complexity,  relational 
databases  only  allow  for  the  efficient  stor¬ 
age  and  retrieval  of  fixed-record-length, 
alphanumeric  data.  The  proliferation  of 
rich,  and  more  complex  digital  data  types 
(non-fixed  length  records,  containing  data 
other  than  alphanumeric)  led  to  the  devel¬ 
opment  in  the  early  1980s  of  object-ori¬ 
ented  DBMSs  (ODBMSs).  While  allow¬ 
ing  for  more  complex  data  types,  and  be¬ 
ing  entirely  extensible  with  respect  to  data 
types,  ODBMSs  do  not  have  an  effective 
method  for  querying  the  data — there  is  no 
SQL  equivalent. 

However,  there  are  many  additional 
benefits  of  ODBMS  that  have  to  do  with 
systems  development  issues,  which  use 
the  inheritance  and  encapsulation  nature 
of  “objects”  in  object-oriented  programs. 
The  elements  of  object-oriented  database 
management  systems  represent  very  effi¬ 
cient  building  blocks  for  other  applications. 


Inefficient  Method  of  Combining  the 
Advantages  of  RDBMS  and  ODBMS 


Most  relational  database  management 
systems  have  attempted  to  capture  the 
benefits  of  object-oriented  database 
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management  systems  by  placing  an  ob¬ 
ject  layer  on  their  existing  systems.  Al¬ 
though  such  an  approach  preserves  legacy 
applications  (SQL-based),  it  possesses  sig¬ 
nificant  flaws.  Grafting  a  fundamentally 
different  system  to  an  old  architecture 
means  that  the  basic  search  engine  is  un¬ 
able  to  understand  how  to  optimize  stor¬ 
age  and  retrieval  features  of  object  data 
records.  At  best  such  an  approach  will  be 
extremely  inefficient:  slow,  and  requiring 
extra  processing,  storage,  and  communi¬ 
cations  resources.  The  magnitude  of  the 
inefficiency  problem  is  geometrically  pro¬ 
portional  to  data  complexity  and  the  size 
of  the  database(s). 


Object-relational  Database 
Management  System 


A  more  straightforward  and  promising 
approach  is  to  re-architect  a  DBMS  from 
the  ground  up,  using  the  following  meth¬ 
ods: 

•  Incorporate  artificial  intelligence  soft¬ 
ware  with  a  knowledge  of  objects. 

•  Create  “smart  objects.” 

•  Construct  query  languages  (including 
SQL  legacy  systems  as  a  subset). 

•  Redevelop  client/server  architectures. 

•  Integrate  extendibility  of  data  types 
into  the  re-architectured  system  from 
the  beginning. 

Such  an  object-relational  database  man¬ 
agement  system  (ORDBMS)  was  created 
in  the  early  1990s  and  has  been  operation¬ 
ally  tested  in  a  wide  variety  of  applications. 


with  extremely  diverse  sets  of  data  types. 
The  ORDBMS  approach  is  ideally  suited 
to  the  size,  complexity,  extendibility  re¬ 
quirements,  and  the  need  for  flexible  (and 
inherently  unpredictable)  search  strategies 
that  characterize  national  intelligence  da¬ 
tabase  applications. 


Search  Engines 


## 


Finally,  a  key  to  a  superior  solution  to 
the  management  of  intelligence  informa¬ 
tion  is  the  abil¬ 
ity  to  precisely 
identify  indi¬ 
vidual  records 
(data  objects)  of 
extremely  var¬ 
ied  types,  and  to 
collate  them 
from  a  widely 
distributed  set 
of  heteroge¬ 
neous  sources. 

Natural  lan¬ 
guage  has  evol¬ 
ved  over  many 
millennia  as  a 

way  to  describe,  with  precision,  diverse 
kinds  of  objects,  information,  and  abstrac¬ 
tions.  Natural  language  is  adapted  to  the 
interpretation,  perception,  and  correlation 
tasks  of  natural  intelligence.  A  natural  lan¬ 
guage  processing  search  engine  is  a  man¬ 
ageable  approach  to  finding  the  needles 
in  the  diverse,  large,  and  expanding  hay¬ 
stacks  of  potentially  relevant  information. 

Pure  keyword  indices,  whether 
weighted  or  not,  can  provide  a  useful  first 
cut  in  identifying  and  retrieving  user-de¬ 
fined,  relevant  intelligence  information. 
However,  the  failure  to  “understand”  the 


a  key  to  a 
superior  solution  to 
the  management  of 
inteMigence  inf orma< 
tion  is  the  ability  to 
precisely  idlenf ify 
individual  records 
(data  objects)  of 
extremely  varied 
types^  and  to  collate 
them  from  a  widely 
distributed  set  of 
heterogeneous 
sources." 
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content  of  individual  data  records  with  no 
more  depth  than  a  simple  frequency  of 
keyword  mentions  in  a  data  record  high¬ 
lights  two  problems.  First,  too  many  “rel¬ 
evant”  records 

H.r.9.  ..d  r.-  Second,  .f 

trieval  »yileiiis...#or  a  particular  con- 
performance  pur 


poses,  search  and 
retrieval  is  a  two- 
stage  process." 


cept  or  substan¬ 
tive  element  of 
content  is  de¬ 
scribed  in  dif¬ 
ferent  words  in 
several  different  sources  or  records,  many 
relevant  records  will  be  missed.  The  pro¬ 
posed  natural  language  search  engine  has 
the  ability  to  represent  a  particular  unit  of 
text  or  language  as  an  abstraction  of  data 
that  contains  concepts  and  information,  in 
addition  to  including  a  more  sophisticated 
method  for  counting  keywords  within  that 
unit  of  text. 


General  Indexing  and  Search,  and 
Storage  and  Retrieval  Strategies 


Indexing  and  Search  Engines 

In  most  information  storage  and  re¬ 
trieval  systems,  including  several  popu¬ 
lar  Internet  browsers  (retrieval  systems  for 
Web  pages),  for  performance  purposes, 
search  and  retrieval  is  a  two-stage  process. 
First,  all  relevant  records  and  data  ele¬ 
ments  in  storage  are  “read  and  analyzed” 
by  an  application  or  routine  that  could  be 
viewed  as  a  general  purpose  search  engine. 
The  routine  analyzes  each  record  or  file 


and  notices  key  features  of  that  file,  using 
those  features  to  create  an  index  for  that 
file  or  record.  For  free-form  text,  the  in¬ 
dex  might  consist  of  key  words,  perhaps 
weighted  by  their  frequency  or  relative  lo¬ 
cation  in  the  text.  In  more  sophisticated 
natural  language  processing  routines, 
“concepts”  (key  words,  linked  together  in 
a  particular  linguistic  structure),  and 
weights  attached  to  the  relative  importance 
of  key  words  and  concepts  in  the  text 
might  also  be  included.^ 

The  second  stage  is  to  retrieve  a  tar¬ 
geted  subset  of  the  records  represented  by 
the  master  index.  To  do  this,  the  routines 
analyze  a  query  by  the  same  routines  in 
the  same  way  that  individual  records  were 
processed  in  creating  the  master  index,  and 
a  special,  much  smaller  index  is  created 
for  the  query.  The  index  for  the  query  is 
then  compared  with  the  master  index.  The 
result  is  a  detailed  report  of  pointers  to  that 
subset  of  individual  records  where  there 
are  matches  between  the  query’s  index  and 
the  master  index.  Armed  with  this  report, 
the  user  retrieves  those  records  that  look 
particularly  interesting.  The  initial  index¬ 
ing  of  individual  data  records  and  files, 
and  the  search  for  and  identification  of 
individual  records  is  a  very  similar  process. 

In  an  image  understanding  routine,  vi¬ 
sual  content — ^the  structure,  composition, 
texture,  color,  or  object — along  with  an¬ 
cillary  information  (e.g.,  the  source  or 
date)  might  form  the  basis  for  the  image’s 
index.  A  master  index  could  then  be  con¬ 
structed  that  would  be  an  overlapping  set 
or  a  composite  of  all  the  individual  indexes 
for  individual  records.  Nearly  all  existing 


^  The  leadinq  knowledge-based,  natural  language  processing  system  comes  from  the  Center  for  Machine 
Translation^at  Carnegie  Mellon  University  (Jaime  Carbonell,  founding  director).  A  simpler  version  of  the 
core  of  their  machine  translation  system  is  the  basis  of  the  Lycos'"  Web  browser. 


28 


A  Multimedia  Systems  Approach  to  National  Security  Policy^  Decision  Making,  and  Intelligence  Support 


image  and  video  search  and  retrieval  sys¬ 
tems  rely  entirely  on  narrative,  linguistic 
descriptions  to  identify  a  set  of  images  or 
video  sequences  that  are  then  examined 
visually  by  the  user. 

Use  of  Linguistic  Content 


Composite  Search  Engines 

Search  engines  designed  for  use  with  a 
single  data  type  (e.g.,  text,  image,  video 
clip,  or  sensor  output)  may  be  useful  in 
delimiting  human  search  of  more  complex 
or  ambiguous  data  and  content.  The  ques¬ 
tion  remains  as 


TO  Identify  Images  to  whether  one 

Indexing  and  retrieval  of  video  se-  maKe  sev- 

quences  or  “clips”  from  vast  stores  of  search  en-  \,^n,  a  single  data 

video  data,  based  on  content,  implies  that  (one  for  type...  may  be  useful 

the  content  of  the  video  clip  is  used,  di-  delimiting  human 

rectly  or  indirectly,  as  the  basis  for  stor-  tandem  search  of  more 

age  and  retrieval.  In  order  to  capture  as  produce  re-  complex  or  ambigu- 
much  content  as  possible,  the  linguistic  superior  to 

content  of  the  audio  track  is  treated  as  part  search  en-  • 

of  the  video  clip’s  content,  and  in  many  working 

cases,  for  storage  and  retrieval  purposes  alone  (Jennings,  1994).  We  anticipate  that 
the  audio  track  content  is  assumed  to  cap-  answer  is  “yes.”  An  important  step  in 
ture  all  of  the  video  clip’s  content.  creating  the  type  of  flexible  information 

A  leading  technology  in  this  regard  is  management  system  advocated  here  for 
embodied  in  the  Carnegie  Mellon  Univer-  national  security  and  intelligence  use  is 
sity  Informedia  Project.  Speech  recogni-  ^  superior,  composite  search  en- 

tion  software.  Sphinx  II,  is  used  to  auto-  several  “single  data  type”  search 

matically  translate  a  video  clip’s  audio  and  retrieval  systems.  A  composite  search 
track  into  free-form  text.  engine  would,  at  minimum,  include  both 


Language  processing  software  similar 
to  that  used  in  Lycos™  (http:// 
www.lycos.com/),  a  popular  Internet  Web 
page  search  engine,  generates  a  content- 
based  index  for  the  video  clip.  It  is  this 
index  that  forms  the  basis  for  search  and 
retrieval  of  video  clips.  It  should  be  noted 
that  approaches  that  depend  exclusively 
on  natural  language  understanding 
generally  fail  to  sufficiently  categorize  the 
associated  imagery. 

Indexes  and  searches  in  the  approach 
advocated  here  involve  both  natural  lan¬ 
guage  processing  and  image  recognition 
software,  functioning  both  as  indexes  and 
search  engine. 


natural  language  understanding-based  and 
image  understanding-based  search  en¬ 
gines. 

Two  issues,  both  heavily  influenced  by 
the  context  and  knowledge  domain  of  the 
search,  must  be  addressed  as  this  compos¬ 
ite  search  tool  is  developed.  First  is  the 
role  of  ancillary  information  and  data  in 
identifying  appropriate  records.  (For  ex¬ 
ample,  a  military  analyst  might  profitably 
use  historical  time  lines,  place  names,  per¬ 
son  name  files,  and  newspaper  articles  to 
correctly  locate  video  clips  with  particu¬ 
lar  content.)  Second  is  the  purpose  of  the 
search,  which  will  help  determine  relative 
weights  placed  on  the  dimensions  of  the 
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search.  Both  issues  influence  the  choice 
of  the  archive  of  video,  text,  and  other  in¬ 
formation  to  test  various  components  of 
any  composite  search  engine. 

Common  Approaches  to  Storage  and 
Retrieval  of  Video  Sequences 

The  approach  taken  by  the  Carnegie 
Mellon  Informedia™  project  (mentioned 
above)  to  storing  and  retrieving  video 
material,  although  considerably  more  so¬ 
phisticated,^  is  similar  in  broad  outlines 
to  that  employed  by  large  broadcast  news 
organizations.  Both  make  use  of  linguis¬ 
tic  content  descriptions  for  indexing  or 
cataloging  video  sequences  and  digital 
images  and  for  retrieving  them. 

CNN  has  developed  a  system  for  stor¬ 
ing,  cataloging,  and  sometimes  retrieving 
the  hundreds  of  thousands  of  hours  of  raw 
video  feeds  (eight  feeds,  more  or  less  con¬ 
tinuously,  over 


on  a  particular  video  tape,  which  has  been 
catalogued,  much  like  a  book,  and  placed 
on  shelves. 

That  this  imagery  and  its  associated  lin¬ 
guistic  descriptions  or  narratives/tran¬ 
scripts  have  only  somewhat  overlapping 
content  is  demonstrated  by  the  fact  that 
more  than  80  percent  of  the  requests  for 
historical  footage  or  video  clips  made  to 
the  CNN  Library  are  simply  for  images 
or  particular  kinds  of  visual  scenes  and  set¬ 
tings.  Particular  requests  for  historical 
video  clips  generally  have  little  or  noth¬ 
ing  to  do  with  story  content.  What  is  re¬ 
quested  is  a  visual  backdrop  to  an  often 
unrelated  story.  This  approach  is  insuffi¬ 
cient  for  intelligent  analysts  and  national 
security  policy  makers. 

Performance  Characteristics  of 
Composite  Search  Engines 


"None  of  the  lan¬ 
guage  or  image 
processing  compo¬ 
nents  that  make  up 
the  composite  infor¬ 
mation  retrieval  and 
search  engine  will 
be  100  percent 
accurate." 


each  24-hour 
period)  and  the 
5  percent  that 
make  broad¬ 
casts. CNN  has 
a  human  viewer 
who,  as  the 
video  feeds  are 
received  from 
the  field,  di¬ 


vides  them  into  sequences  or  “clips”  with 
a  beginning  and  end,  and  types  brief  notes 
describing  their  contents.  It  is  the  notes 
that  are  searched  (not  the  images  them- 


None  of  the  language  or  image  process¬ 
ing  components  that  make  up  the  compos¬ 
ite  information  retrieval  and  search  engine 
will  be  100  percent  accurate.  An  impor¬ 
tant  question  (once  the  “best  of  breed 
search  engine  components  are  identified) 
is  to  how  reliably  each  component  has  to 
perform  in  order  to  be  good  enough.  Be¬ 
cause  each  component  is  embedded  in  a 
complex  system  and  interacts  with  a  par¬ 
ticular  knowledge  domain,  there  is  no  easy 
answer.  And  the  best  systems  will  exploit 
the  characteristics  of  particular  knowledge 


selves),  via  computer,  pointing  to  a  place  domains. 

3  This  strategy  uses  audio  track  and  speech^recognition  software  to  generate  an  automatic  transcript  that 
presumes  to  describe  content  of  the  video. 

‘  This  description  results  from  extensive  discussions  with  Cable  News  Network,  particularly  with  the  News 
Tape  Librarian. 
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For  example,  the  best  available  speech  free- form  text  search  engine  paired  with 
recognition  software  today  may  be  85  per-  image  recognition  software  and  standard 
cent  accurate  in  speaker-independent  set-  SQL-based  search,  when  coupled  with  re¬ 
tings  with  a  20,000  word  vocabulary,  and  cent  advances  in  processing,  storage,  dis- 
98  percent  accurate  with  an  80,000  word  play,  and  telecommunications  technolo- 
vocabulary,  but  the  larger  vocabulary  sys-  gies,  holds  great  promise  for  the  informa- 
tem  may  cost  16  times  as  much  and  take  tion  management  systems  appropriate  for 
10  times  as  long  to  search.  Video  se-  military  and  national  intelligence  func- 
quences  indexed  using  the  audio  sound  tions.^ 

track  and  85  percent  speech  recognition  Multimedia  Archival  Systems,  Inc., 

may  be  good  enough” — accurately  in-  (MmAS)  is  developing  a  robust  informa- 

dexed  and  retrieved  under  such  conditions  tion  management  system^  to  accommodate 

with  an  elaborate,  detailed  query,  which  the  automatic 

depends  on  the  uniqueness  of  the  clip  or  indexing, 

query  and  the  redundancy  of  the  responses  archiving  and  "The  prototype 

to  it.  Or,  when  coupled  with  image  recog-  retrieval  needs  tiosign  (of  the  infer* 

nition  software  that  is  75  percent  accurate  of  intelligence  management 

on  identifying  video  clips,  the  combina-  analysts  and  na-  ®y**®"*)  employs 

tion  of  search  tools  may  be  such  that  the  tional  security  V®®***  recognition, 

accuracy  of  the  composite  system  is  sub-  policy  makers 

stantially  better  than  the  accuracy  of  ei-  and  decision  {JJI*”** 

ther  component.  Or  the  opposite  may  be  makers  with  di-  technologies.  " 

true,  depending  upon  the  content  and  the  verse  data  needs 


characteristics  of  the  knowledge  domain. 
Information  on  troop  movements  and  real 
time  deployment  is  different  from  infor¬ 
mation  about  national  political  subgroups, 
which  is  different  from  water  resource  in¬ 
formation  or  the  characteristics  of  a 
nation’s  secondary  educational  systems. 


that  remain  un¬ 
derserved  by  current  technological  ap¬ 
proaches.  Integrating  key  portions  of  stan¬ 
dard  Web  browser  technology  (Netscape 
Navigator'T^  and  Microsoft  Internet  Ex¬ 
plorer™)  with  the  capabilities  of  the 
Informix/IllustraTM  ORDBMS,  Multime- 


One  Approach  to  Information 
Management  for  Intelligence  Anaitsts 

The  combination  of  an  ORDBMS  with 
a  natural-language-processing-based, 


dia  Archival  Systems  has  developed  a  pro¬ 
totype  of  a  client/server  system  for  index¬ 
ing,  cross-referencing,  and  retrieving 
data  stored  in  free-form  text,  audio,  and 
video  image  formats,  along  with  stan¬ 
dard  RDBMS  data  types,  to  support  spe¬ 
cific  national  security  decision  making, 


®  The  overall  system  described  has  a  patent  pending  and  uses  existing  and  pending  commercial  licenses 
from  Netscape lllustra"'*  Information  Technologies/Informix,  SUN  Microsystems"^,  and  Carnegie  Mellon 
University,  issued  to  Virtual  University  International,  Inc.,  with  sublicensee  to  MmAS. 

®  MmAS  shares  in  the  technological  developments  of  Virtual  University  International,  Inc.,  which  is  using 
similar  technologies  in  its  university  course  content  preparation  and  delivery  system. 
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intelligence  assessment,  and  military  com¬ 
mand  function  needs.  The  prototype  de¬ 
sign  employs  speech  recognition,  natural 
language  processing,  and  image  recogni¬ 
tion  technologies.  Future  versions  will  al¬ 
low  users  to  create  automatic  cross-refer¬ 
encing  of  data  records  of  different  types. 
The  proprietary  automatic  cross-referenc¬ 
ing  scheme  is  a  kind  of  correlation  coeffi¬ 
cient  or  mea¬ 
sure  of  associa- 
"A  standard  build-  for  eclectic, 

lo-specificalions  “»  «  non^lphanu- 
fixed  price  approach  ^  a 

will  almost  certainly 
fail  to  exploit  the  aids  in  the  syn- 
latest  technology."  Aetic,  intellec¬ 
tual  tasks  com¬ 
mon  to  most  top-level  national  security  and 
intelligence  policy  makers  and  analysts. 

Object-oriented  programming  tech¬ 
niques  create  extensible  systems,  both  in 
terms  of  data  types  and  applications.  The 
Multimedia  Archival,  Inc.,  information 


management  system  uses  COTS  technolo¬ 
gies  and  industry  standards,  and  it  has  the 
common  look  and  feel  of  the  leading 
Internet  browsers. 

The  Internet  protocols  and  design  stan¬ 
dards  form  the  primary  client/server  ar¬ 
chitecture  and  distribution  system  and  it, 
rather  than  the  computer  operating  system, 
functions  as  the  software  development 
platform  for  the  systems  outlined  here.  An 
“Internet-compliant”  strategy  tracks  the 
Internet’s  rapidly  evolving  technological 
advances  and,  in  so  doing,  maximizes  mar- 
kets  and  maintains  compatibility  with  the 
widest  array  of  COTS  hardware,  commu¬ 
nications,  software,  and  information  re¬ 
sources.  This  approach  is  also  referred  to 
as  an  Intranet  system  or  strategy. 

In  the  context  of  national  security  and 


military  intelligence  applications,  the 


system  described  here  has  two  major  func¬ 
tions.  First  is  to  customize  the  MmAS  cli¬ 
ent/server  information  management  sys¬ 
tem  to  the  particular  demands  of  intelli¬ 
gence  and  national  security  data.  This 
means  creating  efficient  natural-language 
and  image-recognition-based  search  en¬ 
gines,  user  interfaces,  and  data  objects  cus¬ 
tomized  for  the  particular  data  types  used 
in  national  security  and  intelligence  sys¬ 
tems.  The  second  is  to  organize  data  cap¬ 
ture  operations  that  will  convert  raw  and 
analog  data— text,  images,  sensor  infor¬ 
mation — into  digital  form,  which  then  can 
be  stored  and  retrieved  efficiently  using 
ORDBMS  server  technology  licensed 
from  Informix/Illustra™  (through 
MmAS). 


IMPLICATIOHS  FOR  THE 

Acquisition  Process _ _ 

if  government  national  security  and  in¬ 
telligence  organizations  are  to  capture  the 
functional  and  economic  benefits  of  the 
rapid  revolution  in  information  technol¬ 
ogy,  it  must  use  COTS  hardware  and  soft¬ 
ware  to  the  greatest  extent  possible,  must 
use  industry  standards  whenever  possible, 
and  must  anticipate  and  follow  trends  in 
technology.  The  task  outlined  in  this  pa¬ 
per  is  a  complex  systems  integration  task 
perhaps  best  performed  by  a  nongovern¬ 
mental  unit  operating  under  a  general, 
task-defined  contract  and  free  to  acquire 
necessary  COTS  hardware  and  software, 
integrating  components  to  meet  the  gen¬ 
eral  needs  of  the  client.  A  standard  build- 
to-specifications  at  a  fixed  price  approach 
will  almost  certainly  fail  to  exploit  the 
latest  technology. 
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Commercial  Applicaiions  of  the 
Underlying  Technologies 


Internet-compliant  MmAS  information 
management  systems  have  applications  in 
any  situation  in  which  the  core  informa¬ 
tion  for  an  enterprise  or  profession  does 
not  fit  the  fixed-record-length,  alphanu¬ 
meric-data  requirements  of  the  standard 
RDBMS  model.  Commercialization  op¬ 
portunities  for  the  information  manage¬ 
ment  system  broadly  outlined  here  have 
been  investigated  for  use  in  national  in¬ 
telligence,  print  and  broadcast  news  me¬ 
dia,  as  the  course  content  preparation  and 
delivery  system  for  virtual  universities, 
and  for  use  by  health  care  providers  (Elec- 
tronic  Clinical  Patient  Records/ 
Telemedicine),  Current  development  ac¬ 
tivity  is  aimed  at  the  education,  distance 
learning,  and  corporate  training  markets. 


Conclusion 


Modem  information  and  communica¬ 
tions  technologies,  while  offering  tanta¬ 
lizing  possibilities  to  military  and  intelli¬ 
gence  professionals,  have  certainly  made 


their  jobs  more  difficult  and  complex.  The 
stove-piping  problem  inherent  in  national 
security  and  national  intelligence  organi¬ 
zations  is  exacerbated  by  current  informa¬ 
tion  and  database  technologies.  Whether 
a  decision  maker  or  analyst  is  trying  to 
understand  context,  identify  missing 
pieces  of  information,  or  make  bets  on 
future  scenarios;  whether  attending  to  spe¬ 
cific  real-time  needs  of  a  military  com¬ 
mander  or  attempting  to  identify  clear 
trends  and  major  long-term  tendencies  of 
a  national  system;  whether  using  the  meth¬ 
ods  of  a  Maigret  or  a  Sherlock  Holmes, 
he  or  she  could  use  a  “technical  fix”  that 
helps  put  together  the  pieces  of  diverse, 
rich  information  together  into  a  coherent 
whole. 

Can  modern  information  technology 
help?  Recent  technological  developments, 
particularly  in  the  software  arena,  hold 
considerable  promise  for  enabling  national 
security  and  intelligence  professionals  to 
better  cope  with  the  information  retrieval 
and  synthesis  tasks  necessary  to  perform 
well  in  what  is  often  an  “overly  rich”  in¬ 
formation  environment.  The  answer  is 
maybe — but  not  if  the  sole  concern  is 
modifying  legacy,  relational  database  sys¬ 
tems. 
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LESSONS  LEARNED 


IDENTIFYING  FAQORS  THAT 
CONTRIBUTE  TO  PROGRAM  SUCCESS 

Maj.  Kenneth  J.  Delano,  USJUF 


Department  of  Defense  acquisition  programs  and  projects  frequently 
experience  cost  overruns,  performance  deficiencies,  schedule  delays,  or 
cancellation.  Often,  a  good  program  manager  using  proven  program 
management  practices  can  mean  the  difference  between  success  and  failure. 
By  surveying  program  managers,  examining  successful  programs  within  DoD 
and  relevant  literature  on  program  management  and  defense  acquisition,  we 
have  identified  factors  that  contribute  to  program  success. 


Department  of  Defense  (DoD)  acqui¬ 
sition  programs  and  projects  fre¬ 
quently  experience  cost  overruns, 
performance  deficiencies,  schedule  de¬ 
lays,  or  cancellation.  U.S.  defense  acqui¬ 
sition  is  arguably  the  largest  “business” 
in  the  world.  Annual  purchases  by  DoD 
of  approximately  $178  billion  exceed  the 
combined  purchases  of  General  Motors, 
Exxon,  and  IBM.  Defense  acquisition  in¬ 
volves  almost  15  million  contract  actions 
annually  and  employs  more  than  165,000 
civilian  and  military  workers  who  man¬ 
age  research  and  development,  procure¬ 
ment,  logistics,  and  support  activities 
(Sammet  and  Green,  1990). 

With  such  a  large  system,  errors  and 
inefficiencies  are  bound  to  occur.  Ex¬ 
amples  are  frequently  reported  in  news¬ 
papers  and  magazines,  which  use  these 
examples  to  illustrate  the  poor  state  of  the 


DoD  acquisition  system.  The  real  impact 
(beyond  the  negative  publicity)  is  on  de¬ 
fense  readiness,  performance,  and  cost 
effectiveness.  Since  World  War  II,  six 
blue-ribbon  commissions  have  studied 
DoD  acquisition  and  recommended  rem¬ 
edies.  Adoption  of  some  of  these  recom¬ 
mendations,  new  regulations,  and  laws  has 
failed  to  alter  the  paradigm  (“Rx  for  Ail¬ 
ing  Procurement  System,”  1990). 

This  study  sought  to  identify  factors  that 
contribute  to  program  success.  The  fac¬ 
tors  identified  can  effectively  improve  the 
current  acquisition  system,  vice  the  mul¬ 
tiple  attempts  to  reform  the  system  itself. 

We  used  two  techniques  to  determine 
these  factors.  First  was  a  survey  in  which 
program  managers  were  asked  to  identify 
factors  they  believed  were  key  to  their 
programs’  success.  We  also  conducted  a 
“factor  analysis”  of  acquisition  literature. 
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Approach  ahd  Methods 


The  approach  used  is  based  on  work  by 
Emory  and  Cooper  (1991),  and  can  be 
summarized  thus: 

•  Define  the  management  question.  (In 
this  case  it  is  “What  are  the  most  sig¬ 
nificant  factors  contributing  to  success¬ 
ful  DoD  acquisition  programs?”) 

•  Identify  research  population  and 
sample  to  be  questioned. 

•  Develop  questionnaire  surveys. 

•  Collect  data. 

•  Analyze  data  collected. 

•  Determine  factors  that  contribute  to 
program  success. 

PopuuTiON,  Questionnaire, 

Data  CouEaioN 

Thirty-two  program  managers  partici¬ 
pated  in  the  survey,  the  text  of  which  is  in 
Appendix  A.  Survey  responses  were  un¬ 
signed  and  untraceable  as  to  the  respon¬ 
dent.  Each  respondent  was  provided  with 
a  blank  copy  of  the  survey  and  a  self-ad- 
dressed  stamped  envelope. 

The  questions  were  developed  to  help 
the  respondents  identify  what  they  felt 


were  key  factors  of  program  success.  Ar¬ 
eas  addressed  include  program  manage¬ 
ment,  personnel,  resources,  and  user  re¬ 
quirements.  These  factors  were  used  in  a 
literature  analysis  to  identify  those  that 
contribute  the  most  to  program  success. 

The  survey  results  were  analyzed  to 
select  key  factors  considered  important  by 
the  respondents.  These  factors  were 
ranked  in  order  of  importance  and  catego¬ 
rized  into  common  subject  areas. 

With  data  from  the  survey  results,  we 
applied  McFarland’s  (1992)  factor  analy¬ 
sis  technique.  This  technique  measures  the 
occurrence  of  key  factors  in  a  review  of 
relevant  literature.  The  occurrence  of  a  key 
factor  in  each  article  is  noted.  In  a  repre¬ 
sentative  sample  of  literature,  one  can  de¬ 
termine  the  relative  importance  of  each 
key  factor  to  the  subject  by  noting  and 
comparing  the  number  of  occurrences.  We 
ranked  those  occurrences  in  order  of  fre¬ 
quency  and  by  subject  area. 

With  these  results  in  hand,  we  com¬ 
piled  a  list  of  factors  that  contribute  to 
program  success.  Those  mentioned  most 
frequently  (as  determined  by  factor  analy¬ 
sis)  were  the  ones  examined  for  identifi¬ 
cation  as  factors  of  success.  Qualitative 
indicators  such  as  the  degree  of  applica¬ 
bility  and  history  of  success  of  each  fac¬ 
tor  were  considered. 


Mai  Kenneth  J.  Delano,  USAF,  is  a  student  at  Air  Command  and  Staff  Coiiege.  Prwiously,  he 
was  Chief,  C4I  Training  Requirements  at  Headquarters  Air  Education  and  Training  Command. 
He  has  more  than  13  years  acquisition  experience,  inciuding  8  years  working  in  prograrn  o  - 
ficeHnTte  a  graduate  of  DSMC’s  APMC  98-1.  He  received  his  M.S.  degree  in  technology 
management  from  the  University  of  Maryland  and  his  B.S.  degree  in  electrical 
from  Texas  A&M  University.  He  is  Acquisition  Corps  certified  in  program  management,  systems 
planning  research  development  and  engineering,  communications  and  computers,  and  test. 
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Results  _ 

Survey  Results  and  Analysis 

Eighteen  surveys  were  returned — a  56 
percent  rate  of  return.  This  response  rate 
is  judged  sufficient  to  validate  the  survey 
results.  (One  program  manager  cautioned 
against  expecting  a  high  response.  “We  are 
all  surveyed  out/’  he  said,  explaining  that 
program  managers  are  frequent  targets  of 
official  and  unofficial  surveys.) 

The  responses  to  the  first  survey  ques¬ 
tion  “Please  rank  the  following  in  order 
of  their  importance  as  indicators  of  pro¬ 
gram  success:  meets  initial  operational 
capability  date,  meets  technical  perfor¬ 
mance  objectives,  meets  logistics  support- 
ability  objectives,  works  well  when 
fielded,  meets  cost  objectives,”  are  pre¬ 
sented  in  Table  1. 

Two  factors,  technical  performance  and 
actual  performance  (works  well  when 
fielded)  were  deemed  most  important.  By 
averaging  the  ranking  from  all  responses, 
technical  performance  was  first.  Judged 
least  important  were  meeting  logistics  sup- 
portability  objectives  and  meeting  cost 


objectives. 

Answers  to  the  second  survey  question 
(“List  the  factors  you  believe  contribute 
to  the  success  of  a  program  or  are  indica¬ 
tive  of  program  success”)  fell  into  eight 
categories: 

•  total  team  concept; 

•  program  manager  skills; 

•  program  manager  responsibility  and 
authority; 

•  well-defined  requirements; 

•  stability; 

•  quality  people; 

•  adequate  staffing;  and 

•  acquisition  strategy. 

Program  managers  were  next  asked 
whether  they  were  helped  or  hindered  by 
the  user,  support  agencies,  higher  com¬ 
mands,  Congress,  and  the  General  Ac¬ 
counting  Office  (GAO).  They  responded 
that  user  involvement  and  input  helped 
their  program,  and  that  for  the  most  part. 


Table  Survey  Results 


Rank  of  Program  Success  Indicators 


Most 

Important 

Least 

Important 

Average 

Rank* 

Program  Success  Indicator 

12.5% 

31 .3% 

3.1  (3) 

Meets  Initial  Operational  Capability  Date 

37.5% 

6.2% 

2.2  (1) 

Meets  Technical  Performance  Objectives 

0% 

31.3% 

4.2  (5) 

Meets  Logistics  Supportability  Objectives 

37.5% 

18.7% 

2.4  (2) 

Works  Well  When  Fielded 

12.5% 

12.5% 

3.1  (4) 

Meets  Cost  Objectives 

*(1  =  highest) 
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they  viewed  involvement  from  support 
agencies  and  higher  commands  as  a  hin¬ 
drance,  All  those  who  responded  felt  that 
involvement  of  Congress  and  the  GAO  in 
specific  programs  was  a  detriment  to  pro¬ 
gram  success.  Table  2  presents  their  re¬ 
sponses. 

Finally,  program  managers  were  asked 
to  rank  a  list  of  program  success  factors 
identified  from  a  preliminary  literature  re¬ 
view  using  a  Likert  scale  (1,  not  very  im¬ 
portant;  2,  somewhat  important;  3,  impor¬ 
tant;  4,  very  important;  5,  critical).  The  fac¬ 
tors  and  results  are  presented  in  Table  3. 

The  respondents  felt  that  program  man¬ 
ager  communication  and  leadership  skills 
were  important.  They  also  felt  that  the 
type  and  quality  of  people  assigned  to  sup¬ 
port  the  program  was  important,  as  was  a 
good  relationship  with  the  user  organiza¬ 
tion.  Adequate  resources  and  stability  (re¬ 
quirement,  design,  and  funding)  were 
judged  to  be  next  in  level  of  importance. 
They  did  not  believe  that  the  degree  of 
technical  difficulty  (low  or  high)  of  the 
program  affected  program  success.  The 
results  also  indicated  that  the  program 
manager’s  technical  ability  or  use  of  a 
total  quality  management  program  were 


not  considered  to  be  very  important  to  pro¬ 
gram  success. 

Literature  Factor  Analysis 

The  factor  analysis  technique 
(MacFarland,  1992)  measures  the  occur¬ 
rence  of  key  factors  in  a  survey  of  litera¬ 
ture.  The  occurrence  of  a  key  factor  in  each 
article  is  noted.  In  a  representative  litera¬ 
ture  sample,  the  relative  importance  of 
each  key  factor  to  the  subject  area  can  be 
inferred. 

By  comparing  the  occurrences  of  a  key 
factor  in  a  number  of  articles  against  oc¬ 
currences  of  other  key  factors  in  the  same 
articles,  one  can  calculate  the  relative  im¬ 
portance  of  each  factor .  For  example,  if  a 
key  factor  is  mentioned  in  5  out  of  10  ar¬ 
ticles,  it  has  an  importance  of  50  percent 
for  comparison  purposes.  This  figure  can 
be  compared  to  those  calculated  for  other 
factors  and  conclusions  drawn  as  to  the 
importance  or  emphasis  the  literature 
places  on  each  factor. 

The  results  from  the  first,  second,  and 
fourth  survey  questions  were  used  in 
developing  factors  used  in  the  literature 
review.  The  factors  were  divided  into 
two  main  areas:  acquisition  factors  and 


Table  2.  Survey  Results 


Has  Program 

Success  Been  Helped  or  Hindered? 

Factor 

Helped 

Hindered 

User 

71% 

29% 

Support  Agencies 

20% 

80% 

Higher  Commands 

20% 

80% 

Congress 

0% 

100% 

General  Accounting  Office 

0% 

100% 
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Table  3.  Survey  Results 


Program  Success  Factors  Rank  By  Importance 

Average  Score 

Program  success  factor 

4.42 

Program  manager’s  ability  to  communicate 

4.25 

Type  and  quality  of  people  associated  with  program 

4.25 

Program  manager’s  ability  to  lead 

4.25 

Good  relationship  with  the  user  organization 

4.17 

Resources:  People,  facilities,  money 

4.08 

Product  requirements  and  design  stability 

3.91 

Funding  stability 

3.83 

Good  relationship  with  the  prime  contractor 

3.58 

Program’s  acquisition  strategy 

3.58 

Program  manager’s  acquisition  experience 

3.25 

Program  personnel  continuity 

3.00 

Program  manager  continuity 

3.00 

High  degree  of  technical  difficulty 

2.92 

Program  manager’s  field  experience 

2.67 

Program  manager’s  technical  ability 

2.33 

Total  quality  management  program 

2.25 

Low  degree  of  technical  difficulty 

1  =  Not  very  important 

2  =  Somewhat  important 

3  =  Important 

4  =  Very  important 

5  =  Critical 

resource  factors.  Tables  4  and  5  give  the  and  program  management.  The  highest 

results  of  the  literature  survey,  with  each  correlation  between  a  factor  and  the  lit- 

source  denoted  by  a  letter.  Table  6  lists  erature  reviewed  was  47  percent.  A  tie 

the  correspondence  to  the  actual  source  occurred  between  three  factors:  quality 

in  the  bibliography.  people,  well-defined  requirements,  and 

The  literature  factor  analysis  reveals  that  acquisition  strategy.  All  had  a  correlation 

there  is  a  broad  range  of  subject  matter  of  47  percent.  This  reflects  the  fact  that 

within  the  general  topic  of  acquisition  literature  articles  frequently  focus  on  these 
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Table  5*  Resource  Factor  Analysis 
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Table  6.  Correspondence  Between  Codes  and  Sources 


A 

Baumgartner,  Brown  and  Kelley 

B 

Beltramo 

C 

Clay 

D 

Gansler 

E 

Gregory 

F 

Heberling  and  Graham 

G 

Hicks,  Rich,  Wertheim  and  Meyer 

H 

Hirsch  and  Waelchli 

1 

Kish 

J 

Lesser 

K 

Nelson 

L 

Price  and  Valentine 

M 

"Rx  for  Ailing  Procurement  System” 

N 

Sammet  and  Green 

0 

Settlemyer 

P 

Snoderly  and  Acker 

Q 

Total  Quality  Management  Master  Plan 

R 

Weiss 

S 

Zairi 

Please  consult  References  for  complete  citation. 

important  aspects  of  acquisition  and  pro¬ 
gram  management. 

Because  the  highest  correlation  was  47 
percent,  the  degree  of  significance  was 
calculated  by  using  47  percent  as  the  maxi¬ 
mum.  Factors  with  correlations  between 
32  percent  and  47  percent  were  consid¬ 
ered  to  be  the  most  significant.  Factors 
with  correlations  between  17  percent  and 
3 1  percent  were  judged  moderately  sig¬ 
nificant.  Factors  with  less  than  17  percent 
correlation  were  considered  to  be  least 
significant. 


Discussion 


Elements  of  program  success  were 
identified  by  surveying  DoD  program 
managers.  A  factor  analysis  was  per¬ 
formed  using  relevant  literature.  Those 
factors  with  the  highest  correlation  be¬ 
tween  survey  and  literature  content  were 
identified  as  those  contributing  to  program 
success.  This  two-step  method  provides  a 
means  of  cross-checking  survey  results 
with  current  literature  works  and  focus¬ 
ing  on  factors  that  are  considered  impor¬ 
tant  to  program  success  by  both. 

The  most  significant  elements  contrib¬ 
uting  to  program  success,  as  identified  in 
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the  factor  analysis  of  the  literature,  fall  into 
two  categories.  The  first  is  acquisition  fac¬ 
tors,  which  consist  of: 

•  well-defined  requirements; 

•  the  acquisition  strategy; 

•  a  program  product  that  works  well 
when  fielded;  and 

•  stability  in  the  program. 

The  second  category  we  define  as  re¬ 
source  factors: 

•  quality  people; 

•  program  manager  responsibility  and 
authority; 

•  total  team  concept;  and 

•  program  manager  skills. 

These  factors  also  ranked  high  in  the 
survey  of  program  managers.  This  high 
degree  of  correlation  with  the  survey  adds 
validity  to  MacFarland’s  analysis  tech¬ 
nique.  From  these  results  comes  the  fol¬ 
lowing  list  of  factors  that  contribute  to  pro¬ 
gram  success. 

Well-Defined  Requirements 

A  requirement  is  a  formal  description 
of  a  desired  operational  capability.  Prod¬ 
uct  stability  depends  on  realistic  require¬ 
ments  and  minimizing  changes.  As 
Baumgartner,  Brown,  and  Kelley  (1984) 
state,  systems  that  have  problems  are  usu¬ 
ally  those  that  have  many  changes  during 
design  and  production,  especially  changes 
driven  by  the  user.  Thus  users  should  take 
care  to  avoid  overstated  requirements  that 
delay  production  and  lead  to  higher  costs. 
And  producers  must  carefully  coordinate 


with  the  users  to  ensure  that  the  require¬ 
ments  are  understood  and  well  defined. 

Hicks,  Rich,  Wertheim,  and  Meyer 
(1991)  believe  that  not  enough  time  and 
attention  are  paid  to  successful  programs 
that  could  serve  as  possible  models  for  the 
future.  They  note  that  the  GAO,  which  sel¬ 
dom  compliments  the  defense  acquisition 
process,  identified  the  Navy’s  Fleet  Bal¬ 
listic  Missile  program  as  one  such  highly 
successful  program  spanning  15  years. 
The  GAO  identified  open  dialogue  be¬ 
tween  the  program  manager  and  the  prime 
contractor  and  continuous  communica¬ 
tions  with  the  ultimate  users  as  reasons  for 
this  success. 

Aquisition  Strategy 

Like  any  business  strategy  or  strategic 
plan,  the  acquisition  strategy  is  situation 
and  resource  dependent.  The  program 
manager  should  examine  the  internal  and 
external  environment  to  gauge  resources 
and  support ! 

available.  The  any  business 

acquisition  strategy  or  strategic 
strategy  should  plan,  the  acquisition 
be  crafted  to  strategy  is  situation 

help  further  the  and  resource 
program  objec-  :'i!:.:.^®P®hdept^^ 
tives,  while 

meeting  constraints  placed  upon  the 
program  by  external  regulators  and 
regulations, 

Snoderly  and  Acker  (1981)  cite  one 
strategy  used  to  reduce  acquisition  time 
and  costs.  The  Defense  Support  Program, 
which  produces  ballistic  missile  early 
warning  satellites,  had  a  requirement  to 
purchase  four  satellites  from  their  sole- 
source  contractor,  TRW,  over  a  five-year 
period.  Normally,  the  four  satellites  would 
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be  separately  funded,  purchased  individu¬ 
ally,  and  programmed  for  delivery  in  suc¬ 
ceeding  years.  Parts  and  material  pur¬ 
chases  for  each  satellite  would  be  made 
separately.  Administrative  costs  and  po¬ 
tential  part  obsolescence  costs  would  also 
be  high.  Assembly  and  test  production 
gaps  would  be  created  due  to  uneven  pro¬ 
duction  and  funding. 

The  acquisition  strategy  actually  pur¬ 
sued  offered  cost  savings  of  $134  million 
for  the  procurement.  The  program  office 
acquired  parts  for  all  four  satellites  at  one 

time,  in  more 

"Just  as  if  Is  difficult  economic  quan¬ 
to  hit  a  moving  tides.  A  single 

target,  it  is  difficult  qualification 

to  manage  a  pro-  test  for  all  four 

gram  that  lacks  J  satellites,  be- 

stability;"  cause  of  conti¬ 

nuity  of  design 
and  production,  also  contributed  to  effi¬ 
ciency,  The  above  efforts  resulted  in  de¬ 
livery  of  the  last  satellite  one  year  early, 
saving  program  administration  costs.  The 
acquisition  strategy  was  approved  because 
the  program  was  well  established,  with 
validated  requirements  and  little  chance 
of  change  or  cancellation. 

Works  Well  When  Fielded 

The  ultimate  test  and  determinant  of  the 
success  of  a  program  is  if  the  item  pro¬ 
cured  works  well  in  the  environment  and 
achieves  its  mission.  Delays  in  procure¬ 
ment  or  cost  overruns  are  temporary  prob¬ 
lems  that  must  be  managed  in  order  to 
keep  the  program  alive.  Those  problems 
are  forgotten  once  the  system  is  fielded. 
The  main  concern  of  the  ultimate  user  is 
if  the  system  works  well  when  deployed. 


Stability 

Just  as  it  is  difficult  to  hit  a  moving  tar¬ 
get,  it  is  difficult  to  manage  a  program  that 
lacks  stability.  Changes  in  requirements, 
budgets,  and  resources  make  program 
planning  and  execution  difficult.  The  pro¬ 
gram  manager  must  act  to  maintain  sta¬ 
bility  where  possible  and  manage  change 
where  stability  is  not  possible.  The  pro¬ 
gram  manager,  as  the  prime  program  ad¬ 
vocate,  must  act  to  lay  the  groundwork  for 
external  support  that  will  help  maintain 
stability. 

Clay  (1990)  believes  that  instability  is 
at  the  core  of  most  defense  acquisition 
problems.  He  suggests  five  conditions  for 
creating  stability: 

•  Managers  should  set  out  a  few  key  sys¬ 
tem  objectives,  consistent  with  strate¬ 
gies  and  user  needs  that  are  correctly 
identified  and  held  constant. 

•  Realistic  cost,  schedule,  and  perfor¬ 
mance  estimates  must  be  set.  He  de¬ 
fines  “realistic”  as  the  probability  of 
overperformance  being  equal  to  the 
probability  of  underperformance. 

•  Trained  and  experienced  personnel 
must  be  assigned  to  the  program  who 
can  to  achieve  the  program  objectives. 

•  Resources  approved  and  promised  dur¬ 
ing  the  planning  phase  must  be  pro¬ 
vided,  unless  the  program  fails  to 
achieve  its  objectives. 

•  Commitments  to  complete  acquisition 
tasks  must  be  fulfilled. 

Hirsch  and  Waelchli  (1989)  equate  pro¬ 
gram  stability  with  quality  expert  W. 
Edwards  Demming’s  “constancy  of 
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purpose”  and  state  that  it  has  long  been 
recognized  as  perhaps  the  single  most  im¬ 
portant  contributor  to  efficiency  and  ef¬ 
fectiveness  in  acquisition.  Often  external 
factors  that  the  program  manager  has  little 
or  no  control  over  affect  program  stabil¬ 
ity.  As  the  program’s  top  advocate,  the  pro¬ 
gram  manager  can  still  demonstrate  con¬ 
stancy  of  purpose  even  in  these  situations. 


the  user  and  contractor  serve  to  foster 
communication  and  a  joint  approach  to 
identifying  and  solving  problems.  Al¬ 
though  total  quality  management  did  not 
rank  as  a  success  factor,  the  total  team  con¬ 
cept  is  one  element  of  a  total  quality  man¬ 
agement  program  that  has  been  identified 
as  contributing  to  the  success  of  a  pro¬ 
gram. 


Quality  People 

Well-educated  and  trained  people  are 
essential  to  the  success  of  a  program.  A 
reasonable  amount  of  personnel  stability 
and  continuity  is  desirable.  Successful  pro¬ 
gram  managers  hire  or  develop  a  talented 
workforce,  mold  them  into  a  cohesive 
team,  and  motivate  them  to  help  further 
program  objectives. 

Program  Manager  Responsibility 
AND  Authority 

The  program  manager  is  responsible  for 
the  success  or  failure  of  the  program,  yet 
there  are  many  factors  beyond  his  control. 
The  solution  is  for  the  program  manager 
to  assume  the  authority  commensurate 
with  the  responsibility  for  ensuring  pro¬ 
gram  success.  Baumgartner  et  al.  (1984) 
quote  one  program  manager  as  stating, 
“Any  program  manager  has  as  much  au¬ 
thority  as  he  is  willing  to  step  up  and  take.” 

Total  Team  Concept 

The  program  manager  should  create  a 
program  office  team  atmosphere  where 
everyone  must  work  toward  program 
goals  and  aggressively  manage  the  pro¬ 
gram.  This  team  spirit  promotes  unity  of 
purpose  and  creates  a  corporate  culture 
that  unifies  the  program  office.  Formation 
of  integrated  product  teams  that  include 


^nrhe  ability 
and  skills  of  the 
program  manager 
can  make  or  break 
a  program/' 


Program  Manager  Skills 

The  ability  and  skills  of  the  program 
manager  can  make  or  break  a  program.  A 
combination  of 
leadership  abil¬ 
ity,  communica¬ 
tion  skills,  oper¬ 
ational  back¬ 
ground,  and  edu¬ 
cation  is  impor¬ 
tant.  The  pro¬ 
gram  manager  must  be  able  to  gamer  sup¬ 
port  for  the  program  at  higher  levels,  mo¬ 
tivate  the  team,  and  navigate  the  program 
toward  successful  achievement  of  its 
goals.  The  program  manager — as  the 
program’s  leader  and  manager — is  in 
charge.  Price  and  Valentine  (1992)  recom¬ 
mend  results-oriented  program  manage¬ 
ment  as  an  effective  way  for  program 
managers  to  mold  organizational  culture, 
emphasize  long-term  goals  and  quality, 
and  focus  on  the  big  picture.  Results-ori¬ 
ented  program  managers  have  a  sense  of 
ownership  in  the  program,  believe  in  the 
mission,  and  communicate  this  to  the  pro¬ 
gram  team.  They  create  an  environment 
focused  on  excellence  and  successful  pro¬ 
gram  completion. 

Baumgartner  et  al.  (1984)  consider  the 
ability  of  the  program  manager  to  be  a  vi¬ 
tal  element  in  the  success  of  a  program. 
Successful  programs  have  managers  who 
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have  the  ability  to  communicate  well  with 
all  types  of  audiences,  are  clearly  in 
charge,  take  authority  needed  to  perform 
the  job,  and  hire  quality  people.  The  au¬ 
thors  state  that  one  important  program 
manager  skill  is  dealing  with  the  external 
environment.  They  recount  how  when  one 
program  manager  was  required  to  do 
something  he  disagreed  with,  he  would  ex¬ 
plain  what  the  repercussions  of  that  ac¬ 
tion  would  be.  If  the  person  persisted,  the 
program  manager  explained  that  he  would 
tie  that  person’s  name  to  the  required 
change  and  its  related  cost,  and  schedule 
impacts  so  that  everyone  in  the  program’s 
chain  of  command  would  know  who  was 
behind  that  change.  The  person  usually  ac¬ 
quiesced.  As  one  program  manager  ob¬ 
served,  many  people  in  the  Pentagon  can 
say  no,  creating  problems  for  your  pro¬ 
gram,  but  do  not  have  the  authority  to  say 


yes.  A  program  manager  needs  the  skills 
and  understanding  to  deal  with  the  exist¬ 
ing  acquisition  system  and  bureaucracy, 
and  the  ability  to  adapt  to  changes  in  the 
system. 

Recommendation 


The  factors  contributing  to  program 
success  that  we’ve  provided  here  give  pro¬ 
gram  managers  points  of  reference,  which 
they  should  evaluate  for  inclusion  in  their 
programs.  These  factors  are  widely  rec¬ 
ognized  as  contributing  to  program  suc¬ 
cess  by  their  peers  and  by  the  current  lit¬ 
erature  on  acquisition.  While  all  these  rec¬ 
ommendations  may  not  apply  to  any  par¬ 
ticular  program,  they  provide  an  array  of 
strategies  that  a  manager  can  implement 
and  monitor  to  gauge  their  success. 
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Appendix  A 

Acquisition  Practices  Survey 


1 .  Please  rank  the  following  measures  of  program  success  in  order  of  their  importance: 

_ Meets  initial  operational  capability  date 

_  Meets  technical  performance  objectives 

_ Meets  logistics  supportability  objectives 

_ Works  well  when  fielded 

_  Meets  cost  objectives 


2.  Please  list  the  factors  you  believe  contribute  to  the  success  of  a  program  or  are  in¬ 
dicative  of  program  success: 


3.  Has  the  success  of  your  program  been  helped  or  hindered  by  influences  outside  the 
program  office  such  as: 

Helped  Hindered 

_  _ _  The  user 

_  _  Acquisition  support  agencies 

_  _  Higher  command  headquarters 

_  _ _  Congress 

_  _  General  Accounting  Office 


4.  Please  rate  the  importance  of  the  following  factors  to  program  success  using  this 
scale: 

Not  very  important  1 

Somewhat  important  2 

Important  3 
Very  important  4 
Critical  5 


48 


Identifying  factors  that  Contribute  to  Program  Success 


A.  Type  and  quality  of  people  associated  with  program 

B.  Product  requirements  and  design  stability 

C.  Funding  stability 

D.  Program  manager’s  technical  ability 

E.  Program  manager  continuity 

F.  Program  personnel  continuity 

G.  Program’s  acquisition  strategy 

H.  Resources:  People,  facilities,  money 

I.  High  degree  of  technical  difficulty 

J.  Low  degree  of  technical  difficulty 

K.  Program  manager’s  ability  to  communicate 

L.  Program  manager’s  ability  to  lead 

M.  Program  manager’s  field  experience 

N.  Program  manager’s  acquisition  experience 

O.  Total  Quality  Management  program 

P.  Good  relationship  with  the  prime  contractor 

Q.  Good  relationship  with  the  user  organization 


1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 
1  2  3  4  5 


5.  Please  note  any  comments  you  have  regarding  successful  acquisition  management 
practices  or  this  survey. 
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LESSONS  LEARNED 


INTEGRATING  COST,  EFFEQIVENESS, 
AND  STABILITY 

Dr.  James  P.  Ignizio 

Various  approaches,  ranging  from  conventional  cost-effectiveness  ratios  to 
newer  multiple  objective/multiple  criteria  models,  have  been  used  in  an  attempt 
to  systematically  factor  cost  and  effectiveness  into  acquisition  decisions.  While 
this  is  laudable,  we  believe  that  one  point  has  been  overlooked:  the  critical 
nature  of  the  inherent  stability  of  the  system  to  be  developed  and/or  acquired. 
We  often  hear  about  cost  overruns;  but  what  we  may  not  realize  is  that  the 
underlying  cause  of  such  overruns  may  be  something  as  simple  as  forgetting 
to  consider  the  stability  of  the  system.  In  this  paper  we  attempt  to  indicate  the 
nature  and  impact  of  stability  in  cost-effectiveness  studies — and  to  propose 
topics  for  further  investigation. 


Whether  funding  an  entirely  new 
system  (e.g.,  an  air  defense  sys¬ 
tem)  or  including  a  new  compo¬ 
nent  into  an  existing  system  (e.g.,  inclu¬ 
sion  of  a  new  class  of  helicopter  into  a 
rapid  deployment  system),  the  goal  is  the 
same — ^procure  a  system  that  provides  the 
“biggest  bang  per  buck.”  To  accomplish 
this,  we  generally  seek  a  solution  that 
minimizes  cost  yet  achieves  certain  tar¬ 
get  levels  for  a  variety  of  multiple  mea¬ 
sures  of  system  performance  (e.g.,  range, 
accuracy,  reliability,  weight,  volume, 
survivability,  availability).  Rarely,  if 
ever,  does  one  see  the  inclusion  of  sys¬ 
tem  stability  as  a  measure  of  system  per¬ 
formance.  And  this  may  explain  why 
program  managers  discover  that  their 
system  has  unexpectedly  experienced  a 


significant  cost  overrun — as  a  consequence 
of  its  inherent  instability. 

Before  proceeding  further,  let  us  define 
the  notion  of  stability  itself.  First,  stabil¬ 
ity  analysis  is  not  the  same  thing  as  sensi¬ 
tivity  analysis,  risk  analysis,  or  reliability 
analysis.  Conventional  sensitivity  analy¬ 
sis,  risk  analysis,  or  reliability  analysis  can 
be  effective  for  dealing  with  systems  of 
limited  size  and  complexity,  if  sufficient 
data  (e.g.,  probabilities  of  component  fail¬ 
ure)  exists  to  support  the  needs  of  the 
analysis.  However,  in  many  of  the  real 
world  systems  under  consideration  today, 
problem  size,  system  complexity  and  in¬ 
herent  errors  (or  gaps)  in  data  invariably 
exist.  Furthermore,  these  conventional 
methods  merely  estimate  the  likelihood  of 
the  failure  (or  degradation)  for  a  given 
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system.  As  such,  they  result  in  estimates 
of  such  things  as  MTBF  (mean  time  be¬ 
tween  failures),  or  the  probability  of  a 
particular  failure  mode,  or  the  impact  of 
the  variation  of  a  given  parameter  on  the 
performance  of  a  system. 

Stability  analysis,  on  the  other  hand, 
addresses  the  topic  of  the  “likely  worst 
case  performance”  of  a  system — normally 
a  system  too  large,  complex,  or  messy  (i.e., 
the  typical  real  world  system)  to  be  dealt 
with  effectively  by  more  conventional 
means.  Extensive  experience  (e.g.,  within 
the  aerospace  sector,  the  military  sector, 
and  even  the  financial  sector)  has  shown 
that  a  system  having  a  high  degree  of  reli¬ 
ability  can  and  often  will  fail  as  an  un¬ 
foreseen  consequence  of  the  failure  of  a 
certain  combination  of  components.  Usu¬ 
ally,  the  only  way  to  evaluate  such  poten¬ 
tially  catastrophic  consequences  before¬ 
hand  would  be  to  run  a  brute  force  evalu¬ 
ation  of  all  possible  combinations  of  fail¬ 
ures.  For  any  real  world  system,  such  a 
process  would  take  years,  if  not  centuries, 
on  even  our  most  powerful  computers. 

As  such,  one  can  think  of  stability 
analysis  as  a  systematic  attempt  to  exam¬ 
ine  the  likely  worst-case  performance  of 
a  complex  system.  In  other  words,  it  is  an 
attempt  to  account  for  the  consequences 
of  “Murphy’s  Third  Law:  Anything  that 
can  go  wrong  will.”  In  the  following  ex¬ 
ample,  the  consequences  of  limiting  the 


analysis  of  a  system  to  just  conventional 
performance  measures  is  described — 
and  this  may  be  contrasted  to  the  results 
actually  exhibited  once  a  prototype  of  that 
system  was  actually  constructed  and 
tested. 

Some  years  ago  a  contractor  for  the 
Navy  was  tasked  with  the  job  of  develop¬ 
ing  an  acoustic  array  for  torpedoes.  After 
carefully  determining  the  various  mea¬ 
sures  of  effectiveness  (e.g.,  range,  sensi¬ 
tivity,  sidelobe  suppression,  etc.)  and  the 
cost  elements,  the  contractor — aided  by  a 
group  of  academicians — tried  to  deter¬ 
mine  the  “optimal”  system,  in  terms  of 
cost-effectiveness.  They  constructed  a 
mathematical  model  of  the  system  and 
then  computed,  by  means  of  mathemati¬ 
cal  optimization  tools,  the  solution:  the 
precise  amplitude  and  phase  of  the  acous¬ 
tic  energy  to  be  delivered  to  each  trans¬ 
ducer  in  the  array — accurate  to  the  fifth 
decimal  point. 

The  design  team  went  further  in  their 
analysis  than  is  typical;  performing  a  la¬ 
borious  sensitivity  analysis  of  the  array  as 
a  function  of  changes  in  the  amplitudes 
and  phases  fed  each  array  element.  Since 
the  mathematical  model  was  nonlinear, 
and  since  the  number  of  array  elements 
were  in  the  hundreds,  they  could — of 
course — only  examine  a  finite  and  rela¬ 
tively  limited  number  of  combinations  of 
perturbations  in  amplitude  and  phase. 


James  P.  Ignizio,  Ph.D.,  is  a  professor  of  engineering  at  the  University  of  Virgin!  a  an  da  visiting 
professor  at  the  U.S.  Army  Logistics  Management  College,  Fort  Lee,  VA.  Dr.  Ignizio  is  the  au¬ 
thor  of  seven  books,  more  than  two  dozen  monographs,  and  more  than  250  technical  papers, 
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intelligence,  financial  systems,  and  applied  operations  research.  Dr.  Ignizio  is  a  Fellow  of  HE 
and  recipient  of  the  First  Hartford  Prize. 
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On  paper  the  solution  looked  fabulous, 
Tthe  laboratory  consisted  of  a  water  tun¬ 
nel  in  which  the  array  could  be  tested  in 
what  were  close  to  actual  conditions.  The 
results  for  the  prototype  array  that  had 
been  fabricated  for  testing  weren’t  nearly 
so  good.  In  fact,  they  were  awful.  The 
problem?  A  slight  change  in  some  combi¬ 
nation  of  seemingly  insignificant  changes 
in  two  or  more  attributes  could  result  in  a 
significant  degradation  of  the  array’s  per¬ 
formance.  If  this  array  design  were  actu¬ 
ally  deployed,  it  would  require  the  devel¬ 
opment  of  either  super-sensitive  receiv¬ 
ers  or  a  major  breakthrough  in  manufac¬ 
turing  fabrication  tolerances.  Either  alter¬ 
native  would  add  literally  millions  of  dol¬ 
lars  to  the  cost  of  the  total  system  (trans¬ 
lation:  a  cost  overrun). 

Our  experience,  over  three  decades  of 
system  design  and  cost-effectiveness 
analysis,  indicates  that  systems  designed 
to  be  “optimal”  are  surprisingly  often  more 
likely  to  be  unstable  than  systems  that  are 
more  conservative  (i.e.,  less  “effective”) 
in  design — at  least  on  paper. 

“Optimal”  acquisition  decisions,  no 
matter  how  cost-effective  in  the  conven¬ 
tional  sense,  may  not  be  a  “good  thing.” 


Graphical  Illustrations  or  Instability 


The  reason  system  stability  is  so  im¬ 
portant,  and  why  it  is  so  easily  overlooked, 
may  be  explained  graphically.  Consider  a 
very  simple  problem  in  which  the  perfor¬ 
mance  of  a  system  is  a  (nonlinear)  func¬ 
tion  of  a  single  variable.  The  output  of  this 
system,  as  a  function  of  some  variable  x, 
is  depicted  in  Figure  1.  The  possible  val¬ 
ues  of  X  range  from  0,00  to  0,20.  The 
system  performance  has  two  main  peaks. 


The  highest  is  centered  at  about  0.02  while 
another  peak  (somewhat  lower  but  much 
broader)  is  centered  at  about  0.16. 

Conventional  methods  of  analysis  are 
optimal  seeking.  As  such,  they  would  de¬ 
termine  that  the  optimal  value  of  x  is  found 
at  X  =  0.02  (for  a  performance  value — 
shown  on  the  vertical  axis — of  9  units). 
However,  should  the  performance  func¬ 
tion  be  off  as  little  as  0.01  units  (i.e.,  imag¬ 
ine  a  horizontal  shift  in  the  nonlinear  func¬ 
tion  some  0.01  units  to  the  left  or  right), 
the  system  performance  drops  from  9  units 
to  0. 1  units — a  fall  of  roughly  99  percent ! 

Yet,  had  we  selected  a  less  than  opti¬ 
mal  solution,  at;r  =  0.16,  the  performance 
of  the  system  would  only  vary  between 
8.4  and  8.5  units.  In  fact,  for  a  solution  at 
X  =  0. 16,  the  performance  function  could 
shift  by  0.02  units  (twice  that  which  vir¬ 
tually  destroyed 

the  “optimal”  «Our  experience... 
system),  and  ii,di*||les  that 
Still  result  in  a  systems  that  are 
performance  designed  to  be 

level  between  8  "optimal"  are 
and  8.5  units.  surprisingly  often 


more  likely  to  be 
unstable  than  sys¬ 
tems  that  are  more 
conservative  (iee./ 
less  "eff edive")  in 
design-^at  least  on 


paper 


## 


In  a  problem 
this  simple,  con¬ 
ventional  sensi¬ 
tivity  analysis 
(e.g.,  perturba¬ 
tions  of  vari¬ 
ables  and,  possi¬ 
bly,  the  mapping 

of  the  response  surface)  would  likely  suf¬ 
fice.  However,  for  larger,  more  realistic 
problems  (i.e.,  those  having  numerous 
variables  and  in  which  we  have  little  or 
no  idea  as  to  the  shape  of  the  functions 
involved),  such  an  approach  might  not  be 
practical. 
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Figure  1.  Nonlinear  Response  Function 


In  the  next  section  we  present  a  brief  the  systems  that  are  to  be  considered  for 
overview  of  existing  methods  for  stabil-  real  world  procurement  or  development 
ity  analysis.  In  the  section  following  that,  are  invariably  (highly)  nonlinear.  More 
a  proposal  for  an  unconventional  approach  specifically,  they  are  problems  of  combi- 
to  stability  is  presented.  natorial  complexity.  Second,  even  if  the 

problem  is  linear  (or  a  linear  approxima¬ 
tion  seems  reasonable),  sensitivity  analy- 
INCLUDIHG  Stability  in  Cost-  sis  is  not  the  same  thing  as  stability  analysis. 

Effectiveness  Analysis  Sensitivity  analysis  tells  us  only  the 

- —  ^  range  over  which  some  solution  remains 

Some  would  claim  that  all  one  has  to  optimal  (e.g.,  changes  in  the  basis  of  Un¬ 
do  to  consider  system  stability  is  to  per-  ear  programming  models),  or  that  ranp 
form  a  sensitivity  analysis  on  the  model.  of  model  coefficient  values  over  which  the 
The  problem  with  this  assumption  is  two-  solution  still  satisfies  all  constraints 
fold.  First,  conventional  sensitivity  analy-  (Ignizio  and  Cavalier,  1 994).  That  type  of 
ses  are  intended,  for  the  most  part,  for  information,  in  itself,  tells  us  little  about 
strictly  linear,  continuous,  problems — ^and  the  inherent  stability  of  a  given  solution. 
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Other  analysts  have  suggested  that  we 
“simply  include  solution  stability”  as  an¬ 
other  objective  or  constraint — that  is,  as 
yet  another  measure  of  effectiveness.  That 
would  be  fine  if  we  could  determine  a  way 
in  which  to  capture  solution  stability  as  a 
mathematical  function;  but  there  is  no 
effective  way  of  doing  this. 

Others  would  argue  that  we  use  “per¬ 
turbation  analysis.”  In  perturbation  analy¬ 
sis  we  perturb  the  values  of  various  vari¬ 
ables  and  coefficients  in  a  model,  singly 
and  in  combination,  and  observe  the  effect 
(Fiacco  and  Ishizuka,  1990;  Perlis  and 
Ignizio,  1980). 

For  relatively  small  problems  this  may 
provide  satisfactory  results.  But  for  prob¬ 
lems  more  typical  of  those  faced  in  the 
real  world,  the  amount  of  effort  required 
would  be  truly  enormous.  Consider,  for 
example,  just  a  modestly  sized  problem 
involving  10  constraint  functions,  5  ob¬ 
jective  (measure  of  performance)  func¬ 
tions,  and  20  decision  variables  (e.g.,  the 
value  for  each  of  20  rapid  deployment 
force  weapon  types).  In  this  problem  there 
would  be  some  310  model  coefficients  and 
right-hand  side  values.  If  we  restricted  our 
perturbation  analysis  to  just  the  evalua¬ 
tion  of  changes  in  each  coefficient  and 
each  pair  of  coefficients,  we  would  be  re¬ 
quired  to  perform  nearly  48,000  evalua¬ 
tions — each  of  which  would  require  a 
large  number  of  sub-evaluations  (e.g., 
analysis  of  the  increments  in  change  of 
each  coefficient  or  pair  of  coefficients). 
Even  then  we  will  have  ignored  those 
combinations  taken  three  or  more  at  a 
time.  As  such,  even  this  modest  problem 
could  not  be  adequately  analyzed  by 
means  of  perturbation  analysis. 


Stability  Analysis  Via 
Genetic  Algorithms 


If  conventional  methods  cannot  assist 
us  in  an  evaluation  of  stability  analysis, 
then  unconventional  approaches  should  be 
considered.  Recall  the  acoustic  array  de¬ 
sign  problem  discussed  in  an  earlier  sec¬ 
tion.  The  “optimal”  design  was  unstable — 
too  unstable  to  even  consider.  However,  a 
“less-than-optimal”  design  achieved  the 
desired  stability  without  an  excessive  deg¬ 
radation  in  the  performance  measures  that 
were  promised,  on  paper,  by  the  optimal 
system. 

Less-than-optimal  conservative  designs 
would  appear  to  have  a  greater  likelihood 
of  remaining 
stable.  Instead 

of  being  located  method*  cannof  b« 
on  boundaries  relied  upon  fe  assist 
of  the  solution  us  in  an  evaluation 
space,  or  at  the  ef  stability  analysis, 
extreme  points  then  unconventional 
of  that  space,  «ipproaches  should, 
they  tend  to  be  boliovo,  bo 
found  in  the  in- 
terior  of  the 

space,  and  are  thus  dominated,  or  ineffi¬ 
cient.  Consequently,  any  changes  in  the 
system,  or  any  errors  in  the  data  should 
have  less  impact  on  these  solutions. 

A  preliminary  exploration  of  an  “evo¬ 
lutionary”  approach  to  stability  analysis 
provides  a  practical  means  to  systemati¬ 
cally  analyze  both  the  effectiveness  and 
the  stability  of  a  solution.  In  this  approach 
either  genetic  or  evolutionary  program¬ 
ming  is  employed  to  produce  a  popula¬ 
tion  of  final  solutions. 

Genetic  algorithms  (GA)  pursue  a  “sur¬ 
vival  of  the  fittest”  search  for  optimality 
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(Davis,  1991;  Ignizio  and  Cavalier,  1994; 
Zalzala  and  Fleming,  1997).  Typically, 
they  utilize  a  Boolean  coding  of  the  solu¬ 
tion  variables — and  the  result  is  termed  a 
“chromosome.”  Evolutionary  program¬ 
ming  is  essentially  the  same  thing,  but  uses 
real  numbers  in  coding.  For  example,  the 
chromosome  jc  =  (1  0  1  1  0  0)  may  mean 
that  we  fund  project  1,  3,  and  4 — and  do 
not  fund  projects  2,  5,  and  6.  We  see  then 
that  a  “1”  in  the  :r- vector  indicates  that  the 
associated  project  is  funded,  while  a  “0” 
signifies  an  unfunded  effort. 

Rather  than  starting  from  a  single  solu¬ 
tion  point  and  conducting  an  iterative 
search,  and  then  moving  in  the  direction 
of  local  optimality,  genetic  or  evolution¬ 
ary  programming  begins  with  a  popula¬ 
tion  of  solutions 

-Inherently  unstable  (tens,  or  even 
solutions  de-evolve  hundreds  per 
(the  reverse  of  “generation  ). 

evolution)  to  poor  These  are  evalu- 
solutions  faster  than  ated  for  their 
stable  solutions/'  “fitness”  (i.e., 

the  correspond¬ 
ing  objective  function,  or  functions  are 
evaluated)  and  the  “fittest”  of  the  bunch 
are  considered  for  “mating”  and  “repro¬ 
duction.”  Choices  are  made  stochastically, 
with  preference  given  to  the  most  fit. 
Mated  pairs  may  reproduce,  via  an  ex¬ 
change  of  “chromosomes.”  However, 
since  reproduction  is  also  stochastic,  not 
all  mated  pairs  will  reproduce.  Yet  another 
stochastic  element  is  introduced  in  the  re¬ 
production  process;  some  chromosomes 
may  be  mutated.  These  operations  result 
in  the  population  of  the  next  generation. 
And  the  procedure  continues  until  some 
termination  criterion  is  reached.  The  ref¬ 
erences  provide  further  information  on 
these  notions.  However,  a  detailed  under¬ 


standing  of  GA  is  not  needed  to  appreciate 
the  proposed  procedure. 

While  global  optimality  cannot  be  guar¬ 
anteed,  genetic  and  evolutionary  algo¬ 
rithms  are  effective  in  finding  “very  good” 
solutions.  In  addition,  the  approach  is  in¬ 
herently  parallel,  and  thus  computational 
effort  is  minimized. 

Our  proposal  is  simple  and  straightfor¬ 
ward.  Use  a  genetic  or  evolutionary  pro¬ 
gramming  algorithm  to  solve  the  problem 
under  consideration.  During  the  process¬ 
ing  of  the  algorithm,  maintain  a  file  of  the 
top  10  or  20  solutions  generated  by  the 
algorithm.  At  convergence,  proceed  to  ex¬ 
amine  this  list  of  solutions  for  stability. 

The  question  that  remains,  of  course, 
is  just  how  to  evaluate — in  a  practical 
manner— the  stability  of  a  given  solution. 
Our  approach  is  based  upon  the  follow¬ 
ing  hypothesis:  Inherently  unstable  solu¬ 
tions  de-evolve  (the  reverse  of  evolu¬ 
tion)  to  poor  solutions  faster  than  stable 
solutions. 

Furthermore,  if  a  genetic  algorithm  is 
effective  in  evolving  a  population  of  so¬ 
lutions  toward  a  high  level  of  fitness,  it 
would  seem  reasonable  to  believe  that  it 
would  be  equally  effective  in  de-evolving 
optimal  or  near-optimal  solutions  to  those 
that  are  of  poor  quality.  In  other  words, 
we  will  use  the  GA  on  good  solutions  in 
an  attempt  to  see  how  long  (how  many 
generations)  it  takes  to  de-evolve. 

The  approach  that  we  are  investigating 
may  be  summarized  via  the  following 
steps: 

•  Step  1 :  Model  the  problem  under  con¬ 
sideration  in  a  format  compatible  with 
GA  (e.g.,  code  the  solution  representa¬ 
tion,  determine  crossover,  mutation, 
and  reproduction  operators,  and  pro- 
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vide  a  means  for  the  evaluation  of  the 
fitness  of  each  solution). 

•  Step  2:  Use  a  GA  to  solve  the  prob¬ 
lem;  maintain  a  list  of  the  10  or  20  best 
solutions  generated  by  the  GA. 

•  Step  3:  For  each  of  the  solutions  in  the 
list  of  the  10  or  so  best  solutions,  de¬ 
velop  a  cluster  of  solutions  about  that 
point  by  means  of  perturbation. 

•  Step  4:  For  each  of  the  clusters  in  Step 
3,  use  a  GA  to  search  for  the  least  fit 
solution;  and  record  the  number  of  gen¬ 
erations  required  to  de-evolve  to  a  suf¬ 
ficiently  poor  solution. 

•  Step  5:  Solutions  (i.e.,  the  original  so¬ 
lutions  used  to  develop  the  clusters  of 
Step  3)  that  de-evolve  the  slowest  are 
assumed  to  be  the  most  stable.  Those 
that  de-evolve  fastest  are  assumed  to 
be  least  stable. 

We  should  elaborate  further  on  Steps  3 
and  4.  In  Step  3  we  generate  a  tight  clus¬ 
ter,  about  each  point,  by  means  of  random 
perturbation  of  the  coordinates  of  that 
point — where  each  perturbation  is  held  to 
a  small  size.  In  Step  4  we  could  easily  de- 
evolve  to  a  poor  solution  by  moving  across 
a  constraint  into  an  infeasible  region. 
However,  the  GA  should  be  set  so  as  to 
disallow  such  moves  (e.g.,  assign  any  in¬ 
feasible  point  some  very  high  value).  In 
this  way  the  de-evolution  takes  place  en¬ 
tirely  within  the  (assumed)  solution  space. 

Our  next  step  is  to  perform  small  per¬ 
turbations  about  each  of  these  solutions 
so  as  to  develop  a  “population”  of  solu¬ 
tions  all  in  close  proximity  to  the  original 
solution  point. 


Following  the  development  of  the  clus¬ 


ters  we  would  use 
a  time,  to  gener¬ 
ate  poor  solu¬ 
tions.  Our  inter¬ 
est  is  centered 
about  the  num¬ 
ber  of  genera¬ 
tions  required  to 
de-evolve  to 
some  predeter¬ 
mined  level  rep¬ 
resenting  a  poor 
solution.  Those 


a  GA,  on  one  cluster  at 


"Our  inexf  step  is 
to  pefferni  small 
perturbations  about 
each  of  theie 
solutions  so  as  to 
develop  d  "popula- 
tipn"  of  solutions 
(all  in  close  proxim- 
ify  to  the  original 
solution  point)/' 


clusters  taking  longest  (i.e.,  in  terms  of 
number  of  generations)  to  reach  that  level 
are  assumed  to  be  the  most  stable.  For 
example,  if  it  took  50  generations  for  the 
cluster  about  one  particular  solution  (X) 
to  de-evolve,  and  only  10  for  that  about 
another  (Y),  the  first  solution  (i.e.,  X) 
would  be  considered  a  far  more  stable 
solution. 

How  well  does  this  approach  work?  The 
only  proper  way  to  evaluate  the  method  is 
to  apply  it  to  real  world  problems  with 
real  world  data.  Obviously,  that  is  time- 
consuming.  Thus  far,  we  have  used  the 
approach  on  a  handful  of  actual  prob¬ 
lems  and  found  the  results,  in  each  instance, 
to  be  very  useful  in  predicting  solution 
stability. 


Illustration 


To  illustrate  the  concept,  consider  a 
more  tangible  example:  the  procurement 
of  an  air  defense  system.  Such  a  system  is 
composed  of  a  variety  of  subsystems  and 
components,  and  its  actual  design  is  a 
problem  of  combinatorics.  These  consist 
of  such  things  as  the  choices  of  missiles. 
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missile  warheads,  radar  types,  support  ve¬ 
hicle  types,  and  so  on.  Typically,  the  num¬ 
ber  of  possible  combinations  is  massive. 

A  genetic  algorithm  is  a  highly  effec¬ 
tive  approach  to  problems  of  combina¬ 
torics.  As  such,  we  could  use  such  an 
algorithm  to  generate  not  just  the  “best 
solution,”  but  also  a  list  of  the  top  10, 20, 
or  so  best  solutions  (i.e,,  combinations  of 
subsystems  and  components  forming  the 
air  defense  system).  We  performed  such 
an  analysis  for  the  design  of  a  number  of 
possible  air  defense  systems,  each  differ¬ 
ing  according  to  the  combination  of  sub¬ 
systems  and  components  that  make  up  the 
system.  Using  simulation,  we  then  deter¬ 
mined  the  top  10  candidates  (according 
to  their  effectiveness  to  cost  ratios),  as 
listed  in  Table  1 . 

Our  next  step  was  to  form  tight  clus¬ 
ters  about  each  candidate  solution.  This 
was  achieved  by  perturbing  each  solution 
ever  so  slightly:  that  is,  exchange  one  or 
two  components  in  the  present  solution  for 
components  not  in  solution.  This  exchange 
is  repeated  until  we  have  a  population  of 


solutions  (i.e.,  clusters)  centered  about 
each  candidate  solution. 

We  then  apply  a  GAto  each  of  the  clus¬ 
ters  (one  at  a  time)  and  record  the  number 
of  generations  needed  to  de-evolve.  The 
cluster  taking  the  longest  is  considered  the 
most  stable  population,  and  the  original 
solution  from  which  that  cluster  was 
generated  is  selected  as  the  most  stable 
solution. 

What  we  discovered  was  that  candidate 
system  A  may  have  had  the  “optimal”  ef¬ 
fectiveness  to  cost  ratio,  but  it  was  ex¬ 
tremely  unstable.  Candidate  system  F,  on 
the  other  hand  was,  far  and  away,  the  most 
stable  of  the  10  prototype  air  defense  sys¬ 
tems.  While  its  effectiveness  to  cost  ratio 
may  be  somewhat  lower  than  that  of  A,  it 
likely  makes  up  for  that  deficiency  in 
terms  of  its  superior  stability.  After  all, 
would  the  military  prefer  an  air  defense 
system  that  is  optimal  on  paper,  yet  whose 
performance  can  radically  be  degraded  by 
the  chance  combination  of  failures  for  a 
number  of  components? 


Table  1:  Candidate  Air  Defense  Systems 

Candidate 

Cost 

Effectiveness  Effectiveness/ 

Cost  Ratio 

A 

50 

80.00 

1.6 

B 

48 

76.32 

1.59 

C 

52 

84.42 

1.585 

D 

54 

84.78 

1.57 

E 

49 

76.44 

1.56 

F 

53 

81.62 

1.54 

G 

56 

84.00 

1.5 

H 

57 

79.80 

1.4 

1 

56 

67.20 

1.2 

J 

59 

64.90 

1.1 

58 
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Summary 


In  this  paper  we  have  explored  the  no¬ 
tion  that  cost-effectiveness  analyses,  in 
support  of  the  acquisition  process,  may  be 
ignoring  an  extremely  important,  if  not 
critical  factor:  the  inherent  stability  of  the 
system.  In  such  problems,  stability  may 
be  far  more  important  than  such  conven¬ 
tional,  and  comfortable,  notions  as 
optimality  (i.e.,  the  “best”  cost-effective¬ 
ness).  Yet,  while  debate  rages  as  to  what 
tool  or  tools  are  needed  to  develop  opti¬ 
mal  designs,  relatively  little  attention  is 
paid  to  the  stability  of  solutions  in  what 


will  always  be  imperfect  mathematical 
models. 

We  propose  that  consideration  be  given 
to  the  concept  of  considering  solution  sta¬ 
bility  by  means  of  evolutionary  algo¬ 
rithms.  While  our  hypothesis  (i.e.,  that  in¬ 
herently  unstable  solutions  de-evolve  to 
poor  solutions  much  faster  than  inherently 
stable  results)  has  only  been  explored  at  a 
preliminary  level,  the  results  thus  far  seem 
to  support  the  thesis.  Hopefully,  this  pa¬ 
per  will  encourage  further  efforts  in  this 
area. 
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SUMMARY  RESEARCH  REPORT  ON 
CRITICAL  SUCCESS  FAaORS  IN  FEDERAL 
GOVERNMENT  PROGRAM  MANAGEMENT 

James  H.  Dobbins  and  Ruhard  C,  Donnelly,  Ph.D, 


This  pilot  study  identifies  critical  success  factors  common  to  government 
acquisition  programs.  It  demonstrates  that  how  these  factors  are  identified 
and  used,  how  they  are  measured,  and  how  they  are  influenced  should  be  a 
mandatory  component  of  education  for  every  federal  government  program 
manager  or  program  manager  selectee. 


An  objective  of  program  manage¬ 
ment,  and  of  program  management 
education,  is  repeatable  success  as 
a  program  manager.  It  does  little  good  if 
managers  are  considered  successful  but  do 
not  know  why  they  were  successful  and 
do  not  know  how  to  repeat  their  successes. 
Success  that  is  the  result  of  luck  is  not  re¬ 
ally  success. 

The  use  of  critical  success  factors 
(CSFs)  in  the  management  of  corpora¬ 
tions  has  been  the  subject  of  several 
published  studies.  The  research  we  de¬ 
scribe  here  investigated  CSFs  as  they  ap¬ 
ply  to  acquisition  programs  within  the  De¬ 
partment  of  Defense  (DoD).  The  use  of 
CSFs  in  the  development  of  critical  man¬ 
agement  information  systems  for  the  DoD 
program  manager  would  have  significant 
benefit. 


Although  profit-driven  private  sector 
companies  have  virtual  autonomy  in  their 
selection  of  suppliers  and  partners,  re¬ 
searchers  investigating  CSFs  had  largely 
ignored  federal  government  projects  at  the 
time  this  study  was  done.  Since  then,  a 
few  federal  agencies  have  reported  CSFs 
for  their  organizations,  but  none  of  the 
reports  published  have  indicated  how  the 
data  was  captured  or  validated,  nor  have 
they  provided  a  CSF-based  measurement 
process.  The  activity  in  this  area  within  the 
federal  departments  therefore  seems  to  be 
at  a  fairly  preliminary  stage  of  development. 

The  questions  we  sought  to  answer  in 
this  pilot  study  are: 

•  Are  there  any  general  CSFs  for  DoD 
programs? 
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•  Are  there  any  CSF-related  measures, 
quantitative  or  qualitative,  that  can  be 
used  on  DoD  acquisition  programs? 

•  Are  there  any  significant  differences  in 
the  CSF  data  between  weapon  systems 
and  automated  information  systems 
(AIS)? 


The  Initial  Research  Environment 

The  importance  of  CSFs  in  manage¬ 
ment  first  gained  widespread  attention 
following  publication  of  an  article  by  J.  F. 
Rockart  (1979).  It  showed  the  need  among 
top  executives  for  certain  critical  elements 
of  information,  notg  provided  by  the  man¬ 
agement  information  systems  (MIS)  or  the 
data  analysis  systems  available.  Rockart 
defined  CSFs  as: 

...the  limited  number  of  areas  in 
which  results,  if  they  are  satisfac¬ 
tory,  will  ensure  successful  com¬ 
petitive  performance  for  the  orga¬ 
nization.  They  are  the  few  key 
areas  where  things  must  go  right 
for  the  business  to  flourish.  If  re¬ 
sults  in  these  areas  are  not  ad¬ 
equate,  the  organization’s  efforts 
for  the  period  will  be  less  than 
desired. 


He  further  described  them  as  “areas  of 
activity  that  should  receive  constant  and 
careful  attention  from  management.” 

Rockart  showed  that  executives  suf¬ 
fered  from  data  overload,  but  were  starved 
for  the  right  kind  of  data  essential  to  mak¬ 
ing  the  decisions  necessary  to  manage 
their  enterprises  effectively.  This  includes 
the  identification  of  CSFs  as  well  as  the 
establishment  of  indicators  that  can  alert 
the  executive  when  a  CSF  is  changing  or 
when  the  assumption  upon  which  a  CSF 
is  based  is  no  longer  valid. 

The  initial  Rockart  paper  was  closely 
followed  by  the  publication  of  a  method¬ 
ology  for  CSF  identification  developed  by 
Bullen  and  Rockart  (1981).  The  research 
conducted  since  then  has  been  done  either 
through  the  interview  process  as  described 
by  Bullen  and  Rockart,  or  by  the  ques¬ 
tionnaire  method. 


Critical  Success  Factors 


Identifying  and  managing  CSFs,  and 
tracking  them  separately  from  the  ever- 
increasing  amount  of  data  to  which  execu¬ 
tives  are  subjected,  has  been  the  focus  of 
significant  private  sector  research.  Some 
of  the  research  has  limited  the  study  to 
those  activities  over  which  the  program 
manager  has  direct  control  (Cleland  and 
King,  1988);  the  majority  of  researchers 
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have  broadened  the  focus  to  include  ele¬ 
ments  beyond  the  direct  control  of  a 
project  manager,  but  still  within  the  sphere 
of  things  that  either  he  could  manage,  or 
that  could  exert  significant  influence  on 
his  activities. 

Bullen  (Bullen  and  Rockart,  1981)  has 
suggested  that  CSF  identification  be  fo¬ 
cused  on  whether  CSFs  fall  into  one  or 
more  of  several  key  areas.  These  key  ar¬ 
eas,  plus  one  (modification  management) 
we  have  added,  are: 

•  Global  or  industry  related:  These  are 
activities  essential  to  project  success 
that  would  be  true  of  any  project  or 
company  operating  in  the  particular 
environment  (industry  or  business 
area). 

•  External  influences:  These  CSFs  are 
governed  by  external  factors  that  can 
significantly  influence  the  success  of 
your  endeavor. 

•  Internal  influences:  These  are  deter¬ 
mined  by  internal  factors  that  can  sig¬ 
nificantly  influence  project  success. 

•  Current  and  future:  Included  here  are 
time-driven  CSFs  that  are  essential  to 
project  success.  Current  CSFs  are  ac¬ 
tivities  that  must  be  done  in  the  near 
future.  Future  CSFs  are  those  which  are 
long  range.  Planning  for  the  success  of 
future  CSFs  may  be  an  activity  that  re¬ 
quires  immediate  attention. 

•  Temporal  and  enduring:  These  are  sig¬ 
nificant  influences  that  either  have  a 
short-term  duration  or  are  present 
through  most  or  all  of  a  project. 


•  Risk  abatement:  Some  activities  are 
necessary  in  order  to  avoid  significant 
identified  risks  to  project  success. 

•  Performance:  These  are  identifiable 
levels  of  performance  or  achievement 
that  must  be  realized  for  the  project  to 
be  successful. 

•  Special  monitoring:  These  activities  or 
events  require  special  monitoring,  pro¬ 
tection,  or  contingency  planning  in  or¬ 
der  to  assure  project  success. 

•  Quality:  Quality  requirements,  if  not 
met,  will  mean  the  failure  of  the 
project. 

•  Modification  management:  Some  ac¬ 
tivities  or  conditions  that  currently  ex¬ 
ist  or  are  currently  planned  will,  if  not 
changed,  cause  the  project  to  fail. 

Review  of  the  Literature 


Most  research  has  been  focused  on  the 
identification  of  CSFs  for  executive  level 
managers  in  specific  industries,  or  heads 
of  specific  kinds  of  departments,  princi¬ 
pally  MIS  departments.  There  has  also 
been  some  minimal  research  focused  on 
the  diversity  of  applications  of  CSFs.  One 
fairly  common  problem  with  much  of  the 
reported  research  is  that  many  of  the  iden¬ 
tified  CSFs  have  not  been  stated  in  the 
form  of  an  activity,  as  was  clear  in  the 
original  group  of  definitions  given  by 
Rockart  and  noted  above.  This  led  to  the 
identification  of  CSFs  that  were  ambigu¬ 
ous  and  hard  to  measure. 

One  of  the  early  research  studies  that 
demonstrated  this  problem  was  conducted 


63 


Acquisition  Review  Quarterly— ^Winter  1 998 


by  Boynton  and  Zmud  (1984).  This  re¬ 
search  focused  on  the  use  of  CSF,  and 
showed  that  CSF  analysis  can  be  used  suc¬ 
cessfully  to  identify  key  concerns  of  se¬ 
nior  MIS  management,  can  be  used  in  de¬ 
veloping  strategic  plans,  and  can  help 
identify  critical  implementation  issues. 
CSFs  can  also  be  used  to  help  managers 
achieve  high  performance  and  establish 
guidelines  for  monitoring  a  corporation’s 


activities. 

Boynton  and  Zmud  also  noted  that  CSF 
analysis  demonstrated  certain  weaknesses. 

They  found  that 


"This  research 
focused  on  the  use 
of  CSF,  and  showed 
that  CSF  analysis  can 
be  used  successfully 
to  identify  key 
concerns  of  senior 
MIS  management, 
can  be  used  in 
developing  strategic 
plans,  and  can  help 
identify  critical 
implementation 
issues/' 


CSFs  were  dif¬ 
ficult  to  use  un¬ 
less  analysts 
possess  the  ca¬ 
pability  to  suc¬ 
cessfully  apply 
an  identifica¬ 
tion  process. 
Some  analyst  or 
manager  bias 
may  be  intro¬ 
duced  through 
the  interview 
process,  and  if 
used  as  a  re¬ 


quirements  analysis  methodology  the  re¬ 
sulting  information  model  may  not  accu¬ 
rately  represent  the  deployment  environ¬ 
ment.  But  the  researchers  concluded  that, 
despite  these  criticisms,  the  CSF  method 
generates  user  acceptance  among  senior 
management,  it  works  well  at  the  policy, 
operational,  and  strategic  levels  of  infor¬ 
mation  resource  planning.  It  forms  a 
bridge  between  corporate  strategic  inter¬ 
ests  and  information  systems  (IS)  strate¬ 
gic  planning. 

Boynton  and  Zmud  (1984)  also  found 
that  CSFs  can  help  identify  issues  that 


merit  close  management  attention,  their 
intended  purpose,  and  they  are  useful  for 
requirements  analysis  in  building  concep¬ 
tual  models  of  an  organization  or  a 
manager’s  role.  This  may  not  be  appro¬ 
priate,  however,  where  managers  have 
difficulty  conceptualizing.  Their  data  sup¬ 
ported  the  assertions  that  CSFs  generate 
enthusiasm  from  senior  managers,  im¬ 
prove  user  communications,  and  build 
managerial  support  for  information  tech¬ 
nologies,  and  that  CSFs  were  particularly 
successful  in  defining  organizational  in¬ 
formation  infrastructures.  Their  data 
also  indicated  that  lower-level  manag¬ 
ers  may  have  difficulty  formulating 
meaningful  CSFs  and  specific  informa¬ 
tion  measures.  This  finding  underscores 
the  need  for  specific  training  in  CSF 
identification  and  analysis  processes 
and  the  need  to  describe  these  factors 
in  terms  of  activities. 

Boynton  and  Zmud  (1984)  conclude 
that  the  weaknesses  attributed  to  CSFs  can 
be  overcome  through  careful  application 
of  the  method,  while  CSF  strength  as  a 
structured  design  process  for  eliciting  both 
MIS  plans  and  managerial  information 
needs  is  key  to  its  success. 

In  another  significant  study,  “Variation 
of  Critical  Success  Factors  Over  Stages 
in  the  Project  Life  Cycle”  (Pinto  and 
Prescott,  1988),  the  authors  hypothesized 
a  set  of  CSFs,  and  then  conducted  a  vali¬ 
dation  study  based  on  empirical  evidence. 
The  objective  was  to  identify  a  set  of  CSFs 
for  each  life  cycle  phase  that  were  gen¬ 
eral  rather  than  company-  or  industry-spe¬ 
cific,  and  to  determine  the  relative  impor¬ 
tance  of  the  CSFs  across  life  cycle  phases. 
The  final  set  of  CSFs  were  identified  and 
related  to  the  life  cycles  during  which  they 
were  important  (see  Table  1). 
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Several  identified  CSFs  demonstrate 
the  difficulty  of  not  specifying  the  fac¬ 
tor  in  terms  of  an  activity.  It  is  difficult 
to  measure,  and  therefore  difficult  to 
know  whether  an  activity  has  been  done 
well,  when  it  is  specified  in  terms  as 
ambiguous  as  “technical  tasks”  or  “project 
mission.” 

Zahedi  (1987)  developed  an  evaluation 
of  reliability  of  an  information  system  as 
a  measure  of  the  system’s  success  based 
on  CSFs.  This  research  addressed  the  is¬ 
sue  of  the  difference  between  behavioral 
and  perceived  measures  of  IS  effective¬ 
ness  resulting  from  a  lack  of  conceptual 
foundation  to  guide  proper  measurement 
development,  and  the  absence  of  a  rigor¬ 
ous  program  of  measurement  validation. 
It  identified  the  need  to  define  CSFs  and 
identify  how  they  are  interconnected.  This 
was  another  look  at  a  question  similar  to 
that  investigated  by  Pinto  and  Prescott 


(1988),  but  looking  at  the  set  of  CSFs  from 
a  reliability  viewpoint.  In  each  case  the 
CSFs  were  not  treated  as  isolated  objects 
but  rather  activities  that  are  interrelated. 

In  “The  Multiple  Uses  of  CSFs” 
(Leidecker  and  Bruno,  1984),  the  authors 
stress  the  applicability  of  CSFs  for  strate¬ 
gic  planning  and  business  strategy  devel¬ 
opment,  identification  of  threats  and  op¬ 
portunities,  and  identifying  a  criteria  for 
strengths  and  weaknesses  assessment. 

Walsh  and  Kanter  (1988)  stress  the  im¬ 
portance  of  using  the  CSF  identification 
process  to  identify  major  causes  of  project 
failure  and  then  ranking  these  major 
causes  by  relative  value,  so  that  such  prob¬ 
lems  can  be  avoided  in  future  programs. 

One  of  the  few  comparative  studies 
done  (Chung,  1987)  concluded  that  if  the 
inquirer  wants  to  know  what  management 
is,  then  the  process  view  should  be  stud¬ 
ied.  However,  if  one  wants  to  know  why 


Table  1. 

Importance  of  Various  Critical  Success  Factors  in  the  Project  Life  Cycle 


Phase 

CSF 

Concept 

Project  mission 

Client  consultation 

Planning 

Project  mission 

Top  management  support 

Client  acceptance 

Execution 

Project  mission 

Troubleshooting 

Well-defined  schedule  or  plan 

Technical  tasks 

Client  consultation 

Termination 

Project  mission 

Technical  tasks 

Client  consultation 
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selected  organizations  are  successful  in 
highly  competitive  environments,  then 
one  must  study  the  three  critical  success 
factors  of  corporate  strategies,  human  re¬ 
sources,  and  operational  systems.  His  con¬ 
clusion  is  that  the  truly  successful  com¬ 
panies  deal  with  these  three  CSFs  differ¬ 
ently  from  the  way  they  are  treated  in  other 
companies. 

More  recently,  the  research  has  contin¬ 
ued  with  the  same  commercial  emphasis 
as  described  above,  but  applied  to  current 
business  trends.  One  group  studied  criti¬ 
cal  success  factors  as  they  apply  to  estab¬ 
lishing  strategic  alliances  (Rai,  Borah,  and 

Ramaprasad, 
"CSF  analysis  has  1996).  A  further 
aisp  been  applied  study  of  CSFs 

directly  tP  pepple.  Ip  in  business  alli- 
measure  prpdoctiv-  ances,  this  time 
Ify/^  ^ ^ 

focus  in  the  oil 
and  gas  industry,  was  reported  in  the  trade 
press  (“Seven  Critical  Success  Factors,” 
1996). 

CSF  analysis  has  also  been  used  for 
community  improvement.  This  is  closer 
to  the  public  sector  than  most  studies,  and 
is  an  example  of  the  analysis  being  ap¬ 
plied  to  a  fairly  narrow  focus  area 
(VanDeusen,  1996).  The  researcher 
gleaned  six  factors  from  14  community- 
scale  future  search  conferences  con¬ 
ducted  between  1993  and  1995.  These 
CSFs  are  leadership,  scope,  participation, 
structure,  results,  and  strong  conference 
management. 

Note  once  again  the  ambiguity  and  the 
problem  when  CSFs  are  not  specified  in 
terms  of  activities.  It  is  very  difficult  to 
measure  something  like  “structure”  or 
“scope”  or  even  “leadership,”  especially 
when  something  like  leadership  can 


be  defined  and  measured  in  so  many 
different  ways. 

Business  processes  for  new  product 
development  have  not  escaped  the  appli¬ 
cation  of  CSF  analysis.  A  benchmarking 
research  study  of  161  business  units  (Coo¬ 
per  and  Kleinschmidt,  1996)  identified  the 
CSFs  for  new  product  performance  at  the 
business  unit  level.  The  researchers  found 
that  the  CSFs  fell  into  major  categories. 
Two  key  performance  dimensions— prof¬ 
itability  and  impact— were  identified.  Four 
key  drivers  were  identified:  a  high-quality 
new  product  process,  the  new  product  strat¬ 
egy  for  the  business  unit,  resource  avail¬ 
ability,  and  research  and  development 
spending  levels.  Merely  having  a  formal 
new  product  process  had  no  impact. 

CSF  analysis  has  also  been  applied  di¬ 
rectly  to  people,  to  measure  productivity. 
Christine  Bullen,  one  of  the  leaders  in  the 
application  of  CSF  analysis,  completed  a 
research  study  of  knowledge  worker  pro¬ 
ductivity  (Bullen,  1995).  She  found  that 
the  context-specific  nature  of  personal 
productivity  demands  an  understanding  of 
the  processes  by  which  knowledge  work¬ 
ers  achieve  their  goals  and  objectives. 
Once  the  nature  of  personal  productivity 
is  understood,  measurement  becomes  a 
much  simpler  task  and  the  measures  have 
real  meaning. 

These  studies  all  show  how  CSF  analy¬ 
sis  is  applicable  to  a  wide  variety  of  in¬ 
dustries  and  subsets  of  industries.  CSF 
analysis  has  also  been  effectively  applied 
to  individual  process  areas  within  a  cor¬ 
poration,  such  as  strategic  planning  and 
information  technology  implementation, 
although  it  is  not  routinely  found  as  a  part 
of  strategic  management. 

Research  on  the  application  of  CSFs  to 
program  management,  and  in  particular 
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military  program  management,  is  lacking. 
Research  on  evaluating  the  true  critical¬ 
ity  of  identified  CSFs  is  lacking  in  any  en¬ 
vironment,  government  or  private  sector. 
There  is  an  implied  assumption  in  much 
of  the  research  to  date  that  managers  are 
relatively  equal  in  their  ability  to  iden¬ 
tify  CSFs  that  truly  are  critical  to  project 
success. 


The  Research  Method 


This  research  was  conducted  using  the 
survey  questionnaire  method.  The  survey 
instrument  was  developed  based  on  the 
CSF  categories  identified  by  Bullen 
(1995),  as  noted  above,  plus  the  one  cat¬ 
egory  added  by  these  researchers.  In  the 
instrument,  the  participants  were  asked  to 
identify  both  CSFs  according  to  the  indi¬ 
cated  categories  as  well  as  their  recom¬ 
mended  associated  measures  for  each 
CSF.  The  instrument  was  mailed  to  two 
groups  of  program  managers;  those  man¬ 
aging  embedded  system  programs  and 
those  managing  automated  information 
systems.  The  returns  were  analyzed  and 
the  data  captured  so  that  responses  could 
be  made  to  the  initial  research  questions. 
Data  was  separated  between  the  two 
groups  of  program  managers,  and  the  re¬ 
sults  were  examined  for  each  group  indi¬ 
vidually  as  well  as  comparatively  so  that 
common  CSFs  and  measures  could  be 
identified.  The  findings  were  evaluated  on 
this  basis. 

Having  selected  the  survey  approach, 
the  set  of  program  managers  to  whom 
the  questionnaire  was  sent  was  obtained 
from  lists  maintained  by  the  Information 
Resources  Management  College  (IRMC) 
of  the  National  Defense  University  in 


Washington,  DC,  and  by  the  Defense  Sys¬ 
tems  Management  College  (DSMC)  at  Ft. 
Belvoir,  VA.  Those  from  the  IRMC  were 


program  man¬ 
agers  for  the 
development  of 
non- weapon 
systems,  and  are 
identified  as 
Group  B.  Those 
from  the  DSMC 
list  were  pro-  w 

gram  managers 

for  the  development  of  systems  that  are 
identified  as  embedded  systems,  typi¬ 
cally  weapon  systems,  and  identified  as 
Group  A. 


"Research  on  the 
application  of  CSFs 
to  program  manage¬ 
ment,  and  in  particu¬ 
lar  military  program 
management,  is 
lacking/' 


Initial  Research  Results 


CSF  IDENTIFIWION 

The  data  received  was  broken  down 
into  two  groups.  The  Group  A  universe 
was  73  program  managers.  There  were 
20  Group  A  returns.  The  Group  B  universe 
was  57  program  managers  (14  returns). 

The  returns  were  examined  for  identi¬ 
fication  of  CSFs  and  related  measures  that 
might  be,  or  are  being,  used.  No  one  pro¬ 
gram  manager  reported  more  than  10 
CSFs.  The  Group  A  program  managers 
collectively  reported  37  different  CSFs. 
The  Group  B  program  managers  reported 
29  different  CSFs.  Some  reported  CSFs, 
even  though  they  were  not  stated  in  ex¬ 
actly  the  same  words,  were  similar 
enough  in  wording  that  they  could  be 
combined.  This  distillation  left  us  with  18 
CSFs  common  to  both  groups. 

It  should  be  noted  that  a  few  other 
CSFs  were  identified  in  addition  to  those 
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reported  here.  These  were  very  specific 
to  the  particular  program  and  are  there¬ 
fore  not  included  in  this  report  because  this 
research  effort  sought  to  determine  if  there 
were  CSFs  that  applied  generally  within 
Group  A  and  Group  B,  and  if  there  were 
any  common  to  both  groups. 

Measures 

The  respondents  were  asked  to  identify 
measures  they  use,  or  thought  they  should 
use,  for  each  CSF  they  identified.  There 
was  no  requirement  to  limit  the  number 

of  measures  for 

''The  CSF  idenlifica- 
tion  data  was  eom-  Group  A  re¬ 
plied  and  ranked  spondents  iden- 

by  priorlfy."  tified  41  differ¬ 

ent  measures. 
Each  Group  A  respondent  identified  a  to¬ 
tal  of  at  least  6  measures.  The  Group  B 
respondents  identified  a  total  of  29  differ¬ 
ent  measures.  Three  of  the  respondents  for 
Group  B  identified  no  measures.  One  other 
Group  B  respondent  indicated  that  there 
were  too  many  factors  required  to  come 
together  for  success  and  the  only  real  mea¬ 
sure  is  the  ultimate  outcome. 

Sysvem  Size 

Program  size  was  requested  in  terms  of 
dollar  value  of  the  total  procurement.  The 
responses  received  for  Group  A  indicated 
total  program  sizes  from  $45  million  to 
$30  billion.  The  responses  received  from 
Group  B  indicated  program  sizes  from 
$400,000  to  $3  billion. 

Data  Analysis  Process 

The  CSF  identification  data  was  com¬ 
piled  and  ranked  by  priority.  Each  CSF 
was  given  a  number,  and  the  number  of 


times  each  was  identified  was  recorded. 
Those  CSFs  identified  most  frequently 
were  ranked  highest  in  priority.  This  same 
analysis  was  done  for  both  Group  A  and 
Group  B.  Those  CSFs  identified  as  being 
common  to  both  Group  A  and  Group  B 
were  also  ranked  in  terms  of  frequency  of 
response. 

Measures  identified  by  the  respondents 
are  given  in  a  separate  chart.  The  measures 
reported  were  also  examined  for  their  ap¬ 
plicability  to  the  CSF  identified.  They 
were  also  examined  in  terms  of  their  pri¬ 
ority.  If  a  critical  success  factor  was  seen 
as  high  priority,  we  judged  whether  this 
was  reflected  in  the  measures  identified 
for  that  CSF. 


ANALYSIS  AND  FlHDIHOS  _ 

Table  2  presents  the  CSFs  from  the 
Group  A  responses.  A  unique  component 
of  the  Group  A  response  for  CSF  Number 
4,  “technically  competent  program  office 
staff,”  is  establishing  system  engineering 
expertise  within  the  program  office.  This 
aspect  of  technical  competency  was  not 
present  in  the  Group  B  responses.  Table  3 
shows  Group  B  CSFs.  Note  that  “stable 
and  adequate  budget”  was  the  most  fre¬ 
quently  cited  factor.  Its  prominence  in 
comparison  to  the  other  factors  was  not 
as  dramatic  as  the  most  prominent  CSF  in 
Group  A.  Of  the  1 8  CSFs  common  to  both 
lists,  there  were  evident  differences  in 
emphasis,  both  in  terms  of  frequency  of 
reporting  between  the  two  groups  as  well 
as  the  subtleties  of  their  content  (Table  4). 
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Table  2. 

Group  A  Critical  Success  Factors  in  Priority  Order 

CSF 

No.  Times 

Factor 

12 

15 

Continuous  meaningful  visibility  using  measures 

4 

9 

Technically  competent  program  office  staff 

2 

9 

Clearly  defined  and  stable  requirements,  including  interface 

1 

8 

Stable  and  adequate  funding 

3 

5 

Risk  management 

7 

5 

Schedule  management 

15 

5 

Stable,  qualified  industrial  base 

17 

5 

Effective  vertical  and  lateral  communications 

16 

4 

Management  political  influencing  agents 

6 

3 

Stable  and  adequate  personnel  resources 

8 

3 

Cost  management 

9 

3 

User  involvement,  support,  and  acceptance 

10 

3 

Strong  and  structured  quality  control 

11 

2 

Clearly  and  objectively  defined  project  goals 

19 

2 

Development  and  execution  of  program  management  strategic  plan 

22 

2 

Change  management 

5 

1 

Configuration  management  and  control 

13 

1 

Other  agency  support  for  training  and  government-furnished 
equipment  (GFE) 

14 

1 

Adequate  program  office  resources 

18 

1 

Leadership 

20 

1 

Thorough  system  documentation 

21 

1 

Test  and  evaluation  master  plan  approval 

23 

1 

Program  office  teamwork 

24 

1 

Effective  and  timely  decision  making 

25 

1 

Foreign  military  sales 

26 

1 

Measure  and  control  integrated  logistics  support  performance 

27 

1 

Initiation  of  new  projects 
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fable  3.  ©roup  B  Critical  Success  Factors  in  Priority  Order 


CSF 

No. 

Times 

CSF 

1 

10 

Stable  and  adequate  budget  . . . 

9 

9 

User  involvement  and  support 

12 

9 

Effective  technical  performance  evaluation  _ _ 

2 

8 

Detailed  requirements  analysis  _ _ _ _ _ 

4 

8 

Technically  competent  staff  _ _ _ _ _ 

19 

7 

Top  management  support  .  . . 

17 

6 

Effective  lateral  and  vertical  communications 

7 

6 

Schedule  management  . . . 

10 

6 

Strong  quality  control  program  _ ^  _ 

6 

5 

Stable  proiect  staff 

16 

5 

Management  of  political  influencing  agents  _ _ 

13 

4 

Other  agency  support:  Training  and  GFE  _ _ 

3 

3 

Risk  management  _ _  _ _ 

20 

3 

Strong  knowledge  of  life  cycle  management 

23 

3 

Incremental  acquisition 

8 

2 

Cost  management  _  _ 

22 

2 

Common  sense  _ _ 

11 

2 

Clearly  defined  mission 

14 

2 

Adequate  program  office  resources  _ 

21 

1 

Objective  economic  analysis 

18 

1 

Leadership  . . . . . . 

15 

1 

stable,  qualified  industrial  base  _ _ _ 

5 

1 

Configuration  management  ____ 

24 

1 

On-site  team  to  prevent  fraud,  waste,  and  abuse 

OviRALL  Observations 

Given  the  publicity  generally  afforded 
to  configuration  management,  it  was  a 
surprise  that  this  factor  was  named  only 


once  in  each  group.  This  could  mean  that 
those  reporting  did  not  recognize  the  im¬ 
portance  of  configuration  management,  or 
recognized  its  importance,  but  believe  it 
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Table  4.  SFs  for  Groups  A  and  B  Combined,  in  Priority  Order 


CSF 

No. 

Times 

CSF 

12 

24 

Continuous  meaningful  visibility  using  measures 

1 

18 

Stable  and  adequate  funding 

18 

2 

Leadership 

2 

17 

Clearly  defined  and  stable  requirements,  including  interface 

4 

17 

Technically  competent  program  office  staff 

9 

12 

User  involvement,  support,  acceptance 

7 

11 

Schedule  management 

17 

11 

Effective  vertical  and  lateral  communications 

10 

9 

Strong  and  structured  quality  control 

16 

9 

Management  of  political  influencing  agents 

3 

8 

Risk  management 

6 

8 

Stable  and  adequate  personnel  resources 

15 

6 

Stable,  qualified  industrial  base 

13 

5 

Other  agency  support  for  training  and  GFE  . 

8 

5 

Cost  management  ~ 

11 

4 

Clearly  and  objectively  defined  project  goals 

14 

3 

Adequate  program  office  resources 

5 

2 

Configuration  management  and  control 

is  done  well  enough  now  not  to  be  a  prime 
candidate  for  program  manager  attention. 

The  questionnaire  asked  the  program 
manager  to  list  activities  that  were  be¬ 
lieved  to  be  critical  to  the  success  of  the 
program.  Issues  reported  will,  in  some 
way,  reflect  those  areas  that  have  required 
a  significant  degree  of  program  manager 
attention. 

Another  unexpected  result  was  the 
placement  in  order  of  prominence  in 
which  risk  management  appeared  in  the 


Group  B  list,  being  number  13  in  order  of 
frequency  of  response.  This  same  CSF  is 
ranked  number  6  in  order  of  importance 
in  Group  A. 

With  regard  to  the  CSFs  common  to 
both  groups,  the  list  reflects  a  strong  be¬ 
lief  that  factor  number  12,  continuous 
meaningful  visibility  using  measures,  is 
of  primary  importance  to  program  success 
for  any  system.  Stable  and  adequate  fund¬ 
ing,  clearly  defined  and  stable  require¬ 
ments,  and  technically  competent  program 
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office  staff  were  the  factors  next  in  order 
of  importance,  and  these  three  are  nearly 
equal  in  prominence.  These  fourCSFs  ap¬ 
peared  more  prominently  than  either  cost 
management  or  schedule  management. 
This  may  reflect  a  belief  that  if  these  top 
four  CSFs  are  accomplished,  cost  manage¬ 
ment  and  schedule  management  are  more 
easily  accomplished.  This  may  also  sug¬ 
gest  that  the  focus  of  the  external  over¬ 
sight  groups — cost  and  schedule— is  not 
among  those  activities  that  are  most  im¬ 
portant  to  program  success,  at  least  not  as 
viewed  by  those  responsible  for  execut¬ 
ing  the  mission  of  program  management. 

One  must  also  recognize  that  the  CSFs 
identified  are  not  necessarily  disjoint.  For 
example,  the  factor  continuous  and  mean¬ 


ingful  visibility  will  be  a  necessary  com¬ 
ponent  of  risk  management.  This  is  like¬ 
wise  true  of  the  CSFs  strong  and  struc¬ 
tured  quality  control,  and  technically  com¬ 
petent  program  office  staff.  All  of  these 
tend  to  be  means  for  managing  program 
risk. 


Measures _ _ 

Analysis  of  the  measures  was  accom¬ 
plished  in  light  of  the  identified  CSFs.  It 
was  expected  that  the  most  frequently 
mentioned  CSFs  should  have  measures  re¬ 
flecting  them,  and  those  with  minimal 
mention  may  be  expected  to  have  the  least 
number  of  measures. 


Table  5.  Group  A  Measures  in  Priority  Order 


No. 

Tan 

Frequency 

lie  WriRlip  . . m 

Measure 

1 

10 

Stable  and  adequate  budget  . .  . 

5 

10 

Deviation  from  schedule  .  . 

4 

8 

D eviatio n  from  cost  . 

2 

7 

Number  and  frequency  of  requirements  changes 

6 

7 

Number  of  unique  trouble  reports  _ _  _ _ _ 

3 

4 

Changes  to  the  budget  _ _  _ _ _ _ 

8 

4 

Results  of  tests  of  independent  systems  ^  _ 

20 

4 

Cost  of  change  versus  cost  of  delay  equals  cost  to  improve 

9 

3 

User  or  contractor  walkthroughs  and  reviews  _ _ 

12 

3 

Funding  level  versus  plan 

7 

2 

Number  of  customer  complaints  _ 

10 

2 

Prime  contractor  productivity  per  7000.2 

16 

2 

Program  plan  assessment  (qualitative)  . 

21 

2 

Number  of  reworks  or  rewrites  (continued) 
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Table  5.  Group  A  Measures  in  Priority  Order  (continued) 


No. 

Frequency  Measure 

23 

2 

Time  between  problem  occurrence  and  problem  identification 

25 

2 

Time  delay  of  GFE  deliveries 

1 

1 

Quantitative  assessment  of  requirements 

11 

1 

Program  quality  targets  — ________ 

13 

1 

VROC,  mean  time  between  failures  (MTBF),  Pd,  mean  time 
to  repair  (MTTR) 

14 

1 

System  availability 

15 

1 

Response  to  change  (qualitative) 

17 

1 

Number  of  issues  requiring  higher  approval 

18 

1 

Time  taken  for  approval  decisions 

19 

1 

Number  of  acquisition  protests 

22 

1 

Number  of  first  time  approvals 

24 

1 

Time  to  process  approved  change 

26 

1 

Workload  stability 

27 

1 

Number  and  effect  of  Congressional  interactions 

28 

1 

Effectiveness  of  visibility  processes 

29 

1 

Number  of  delay  or  disruption  claims  from  contractor 

30 

1 

Reject  rates 

31 

1 

Number  of  quality  deficiency  reports 

32 

1 

RAM  measures 

33 

1 

Number  of  miscues  per  month  (no  coordination;  misunderstand) 

34 

1 

Number  of  technical  surprises  per  month 

35 

1 

Number  of  suggestions  adopted  by  contractor 

36 

1 

Number  of  delinquent  action  items— days  late 

37 

1 

Cost  versus  operational  effectiveness 

38 

1 

Number  of  risks  identified,  month  — — — 

39 

1 

Number  of  risks  resolved  per  month 

40 

1 

Number  of  qualified  staff  versus  need 

41 

1 

Number  of  physical  resources  versus  need 
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In  Group  A  (Table  5),  the  most  fre¬ 
quently  mentioned  measure  is  number  5, 
deviation  from  schedule.  The  next  most 
frequently  mentioned  measure  is  number 
4,  deviation  from  cost.  This  may  be  re¬ 
flective  of  the  ease  of  data  collection,  and 
the  need  to  respond  to  the  program  execu¬ 
tive  officer,  the  GAO,  Congress,  and  the 
Defense  Acquisition  Board,  rather  than  fo¬ 
cusing  on  those  activities  that  the  program 
managers  clearly  felt  were  of  significantly 
superior  importance  to  program  success. 

Given  the  importance  afforded  the  CSF 
identified  as  continuous  visibility,  a 
number  of  measures  can  be  considered 


reflective  of  this  CSF,  They  are  shown  in 
Table  6.  In  terms  of  sheer  volume,  these 
measures  reflect  the  importance  afforded 
continuous  visibility.  However,  they  were 
distributed  across  the  spectrum  of  those 
reporting,  and  the  majority  of  the  measures 
were  only  identified  once,  an  obvious 
cause  for  concern.  The  two  most  fre¬ 
quently  mentioned  of  all  the  visibility 
measures  are  those  related  to  requirements 
changes  and  those  related  to  trouble  re¬ 
ports.  The  next  most  frequently  mentioned 
measure  was  number  8,  test  results. 

Walkthroughs  and  reviews,  number  9, 
a  widely  publicized  source  of  visibility. 


Tabic  6* 

Measures  Reflecting  the  Importance  ef  the  CSF  Continuous  Visibility 


No. 

Frequency 

Measure 

2 

7 

Number  and  frequency  of  requirements  changes 

6 

7 

Number  of  unique  trouble  reports  _ _ _ _ 

8 

4 

Test  results  _ 

9 

3 

Walkthroughs  and  reviews  _ 

7 

2 

Number  of  customer  complaints 

10 

2 

Contractor  productivity  . 

1 

1 

Quantitative  assessment  of  requirements 

11 

1 

Program  quality  targets 

13 

1 

VROC,  MTBF,  Pd,  MTTR 

14 

1 

System  availability 

28 

1 

Effectiveness  of  visibility  process 

31 

1 

Number  of  quality  deficiency  reports 

32 

1 

RAM  requirements  _ _ 

34 

1 

Number  of  technical  surprises  per  month 

38 

1 

Number  of  risks  identified  per  month 

39 

1 

Number  of  risks  resolved  per  month 
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Table  7.  Group  B  Measures,  in  Priority  Order 


Frequency 


Measure 


Deviation  from  cost 


Deviation  from  schedule 


4  Number  of  unique  trouble  reports 
3  Program  reviews 
3  Product  quality 
3  System  throughput  (performance) 

2  Number  of  system  requirements  changes 
2  Budget  changes 

2  Funding  level  versus  plan 

2  User  feedback 

2  External  and  internal  Independent  verification  and  validation 

2  Number  of  customer  complaints 

1  Requirements  review 

1  User  acceptance 

1  Test  results  data  reports 

1  Productivity 

1  System  reliability 

1  Downtime:  Rate  and  duration 

1  Progress  demonstration 

1  Contractor  product  demonstrations 

1  Personnel  evaluations 

1  Milestone  resource  review 

1  System  backlog 

1  Analysis  reports 

1  Evaluation  against  oversight  criteria 

1  Number  of  support  complaints 

1  Number  of  software  changes 

1  Time  to  complete  software  change 
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was  only  mentioned  three  times.  This  is 
cause  for  some  concern,  both  because  of 
what  walkthroughs  and  reviews  can  pro¬ 
vide  that  is  not  being  used,  and  because 
of  the  need  to  assure  that  these  visibility 
mechanisms  are  properly  reflected  in  the 
acquisition  process,  particularly  in  the  re¬ 
quest  for  proposal  and  the  contract.  For 
CSF  number  4,  technically  competent  pro¬ 
gram  office  staff  (the  second  most  fre¬ 
quently  mentioned  CSF),  there  are  virtu¬ 
ally  no  measures  reflected. 

It  is  evident  that  there  is  a  lack  of  cor¬ 
relation  between  those  activities  identified 
as  CSFs  for  Group  A  and  therefore  deemed 

critical  to  pro- 


"One  clear  concern 
is  that  if  prograni 
managers  recognize 
certain  activities  to 
be  critical  to  pro« 
gram  success,  and 
the  program  mana¬ 
ger's  information 
network  does  not 
provide  measures 
reflective  of  those 
critical  factors,  then 
their  ability  to  man¬ 
age  those  factors  is 
jeopardized/' 


gram  success, 
and  the  mea¬ 
sures  used  or 
suggested.  The 
number  of  mea¬ 
sures  related  to 
the  top  priority 
CSF,  but  that 
were  only  re¬ 
ported  once,  re¬ 
flects  a  lack  of 
consistency 
among  those 
engaged  in  ac¬ 
quisition  activi¬ 
ties  in  terms  of 


how  commonly  recognized  critical  activi¬ 


ties  can  and  should  be  evaluated. 

In  general  for  Group  B  (Table  7),  the 
lack  of  dominance  of  any  one  measure, 
and  the  deviation  of  the  three  measures 


deviation  from  cost,  deviation  from  sched¬ 
ule,  and  number  of  unique  trouble  reports 
being  the  most  frequently  named,  may  be 
reflective  of  a  general  lack  of  familiarity 
with  quantitative  evaluation  processes  as 
well  as  a  lack  of  consistency  among  those 


engaged  in  non- weapon  system  acquisi¬ 
tion  in  terms  of  how  those  activities 
deemed  critical  to  success  can  and  should 
be  evaluated. 

Although  CSF  number  2,  detailed  re¬ 
quirements  analysis,  is  noted  eight  times, 
measure  number  1,  requirements  review, 
was  only  mentioned  once,  and  measure 
number  2,  number  of  system  requirements 
changes,  was  only  mentioned  twice.  A  to¬ 
tal  of  16  of  the  28  measures  reported  were 
only  mentioned  once.  Six  of  the  measures 
were  only  mentioned  twice.  Therefore, 
only  6  of  the  28  measures  were  mentioned 
more  than  twice,  although  11  of  the  24 
CSFs  for  Group  B  were  listed  at  least  five 
times.  There  is  a  clear  lack  of  correlation 
between  the  activities  considered  critical 
to  program  success  and  the  management 
information  available  by  which  the  pro¬ 
gram  managers  can  measure,  evaluate,  and 
make  use  of  those  critical  factors.  The 
most  frequently  named  measures  are  more 
reflective  of  response  to  oversight  groups 
than  they  are  of  the  issues  actually  con¬ 
sidered  by  the  program  managers  as  most 
critical  to  program  success. 

One  clear  concern  is  that  if  program 
managers  recognize  certain  activities  to  be 
critical  to  program  success,  and  the  pro¬ 
gram  manager’s  information  network  does 
not  provide  measures  reflective  of  those 
critical  factors,  then  their  ability  to  man¬ 
age  those  factors  is  jeopardized.  The  in¬ 
formation  system  used  by  the  program 
office  will  not  be  sufficiently  supportive 
of  the  program  needs. 


Findings 


Based  on  the  above,  we  can  list  a  num¬ 
ber  of  findings. 
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•  The  CSFs  for  DoD  program  manage¬ 
ment  are  identifiable,  and  their  explicit 
identification  would  clearly  assist  the 
program  managers  in  maintaining  man¬ 
agement  focus  on  the  factors  most  im¬ 
portant  to  program  success. 

•  A  significant  number  of  CSFs  are  com¬ 
mon  to  both  Group  A  and  Group  B 
types  of  programs. 

•  The  component  assumptions  and  em¬ 
phasis  for  a  given  CSF  common  to  both 
Group  A  and  Group  B  may  be  slightly 
different.  This  difference  is  largely  a 
function  of  the  difference  between  the 
missions  of  the  two  groups,  Group  A 
being  more  concerned  with  the  com¬ 
plete  development  of  total  systems  than 
is  Group  B. 

•  The  CSFs  identified  by  the  program 
managers  as  the  most  significant  for 
program  success  are  not  those  factors 
that  receive  the  most  attention  from  the 
oversight  activities  and  agencies. 

•  The  measures  identified  most  often  by 
the  program  managers  as  those  used  or 
recommended  are  significantly  more 
oriented  toward  cost  and  schedule 
(which  must  be  briefed  to  the  oversight 
agencies)  rather  than  toward  factors 
identified  by  the  program  managers  in 
the  field  to  be  most  critical  to  the  pro¬ 
gram  success. 

•  There  is  no  widely  recognized  and  gen¬ 
erally  used  set  of  measures  consistent 
with  the  most  frequently  reported 
CSFs.  This  leads  to  the  conclusion  that 
even  though  various  factors  are  recog¬ 
nized  as  critical,  they  are  not  usually 


explicitly  identified  and  the  informa¬ 
tion  network  required  to  manage 
against  those  critical  factors  is  not  de¬ 
veloped. 

•  A  commonly  recognized  set  of  CSFs, 
and  a  consistent  measurement-based 
information  network  based  on  these 
CSFs,  would  be  of  significant  benefit 
to  the  program  managers  as  well  as  the 
oversight  agencies.  Such  a  manage¬ 
ment  system  would  significantly  im¬ 
prove  the  management  success  poten¬ 
tial  on  programs  across  the  board,  and 
would  provide  the  external  groups  a 
consistent  way  of  evaluating  and  com¬ 
paring  different  programs  so  that  rec¬ 
ommendations  for  future  improve¬ 
ments  could  be  intelligently  based. 

•  A  CSF-based  information  network  for 
program  management  would  lend  itself 
to  not  only  increased  visibility  and 
awareness  for  the  program  manager 
and  staff  during  all  life  cycle  phases, 
but  would  provide  the  base  for  the  es¬ 
tablishment  of  measures  for  determin¬ 
ing  when  the  underlying  assumptions 
for  a  given  CSF  may  be  changing. 

•  A  CSF-based  information  network 
would  provide  a  common  framework 
for  productive  discussions  between  the 
program  manager  and  the  external 
groups,  including  the  development 
contractors,  and  would  greatly  support 
the  effectiveness  of  IPTs. 

•  A  CSF-based  information  network 
would  significantly  reduce  the  dupli¬ 
cative  reporting  and  diversions  the  pro¬ 
gram  managers  experience  under  the 
present  conditions. 
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•  A  CSF-based  analysis  process  would  not  respond  at  all,  to  have  a  subordinate 
be  a  significant  teaching  instrument  for  provide  the  responses,  or  to  respond  hast- 
educating  prospective  program  manag-  ily  without  giving  the  matter  the  inte  ec- 

ers  in  strategic  thinking  in  terms  of  tual  time  it  requires. 

those  considerations  critical  to  success. 


Some  Limitations 


The  survey  questionnaire  process  used 
for  this  research  project  has  both  strengths 
and  weaknesses.  An  inherent  strength  of 
the  questionnaire  approach  lies  in  the  num¬ 
ber  of  program  managers  who  can  be  ac¬ 
cessed  simultaneously,  with  the  resultant 


savings  in  time  and  dollars.  It  provides  a 
neutral  and  standardized  method  of  data 
collection  and  allows  those  responding  to 
provide  not  only  the  identification  of 
CSFs,  but  also  a  view  of  the  assumptions 
underlying  the  CSFs  and  the  ways  in 

which  they  can 


"An  inherent 
strength  of  the 
questionnaire  ap¬ 
proach  lies  in  the 
number  of  program 
managers  who  can 
be  accessed  simulta¬ 
neously,  with  the 
resultant  savings  in 
time  and  dollars." 


be  measured. 

One  of  the 
weaknesses  of 
this  process  is 
the  lack  of  op¬ 
portunity  to  dis¬ 
cuss  a  question 
with  the  pro¬ 
gram  manager 
and  draw  out 


responses  that 
fully  consider  the  question  and  its  impli¬ 
cations.  This  can  be  very  important  when 
seeking  responses  from  managers  with 
regard  to  a  concept  or  process  they  are  not 
already  familiar  with.  It  is  subject  to  a  cer¬ 
tain  degree  of  bias  on  the  part  of  those 
responding  and  does  not  allow  for  pur¬ 
suit  of  additional  issues  that  might  be 
important  to  the  result  and  which  might 
not  be  explored  without  the  aid  of  a 
skilled  interviewer.  In  addition,  there  is 


no  protection  from  the  tendency  to  either 


Recommendations 


Based  on  the  information  provided 

above,  we  recommend  the  following. 

•  Educate  program  managers  and  their 
staff  in  the  CSF  identification  process. 
The  failure  to  explicitly  identify  CSFs 
for  a  program  will  invariably  result  in 
the  continued  focus  on  cost  and  sched¬ 
ule  after  they  become  problems,  will 
inhibit  the  development  and  use  of  ef¬ 
fective  life  cycle  measures,  and  will 
prevent  the  development  of  a  truly  ef¬ 
fective  program  management  informa¬ 
tion  system.  Cost  and  schedule  prob¬ 
lems  are  generally  effects,  not  causes. 
They  are  the  results  of  conditions  that 
should  be  identified  and  managed 
much  earlier  than  the  time  when  a  cost 
and  schedule  variance  first  appears. 

•  Educate  the  oversight  agencies  in  the 
CSF  identification  process  and  the  im¬ 
portance  of  this  to  their  management 
functions. 

•  Educate  the  program  managers  and 
their  staff  in  development  of  informa¬ 
tion  networks  consistent  with  the  criti¬ 
cal  success  factors. 

•  Establish  oversight  reporting  mecha¬ 
nisms  consistent  with  critical  success 
factors  so  that  critical  information  is 
reported  when  it  is  needed. 
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Continuing  Research 

Research  is  currently  being  conducted 
using  an  interview  process.  The  results 
will  reflect  whether  there  are  any  changes 
to  the  CSFs  reported  by  program  manag¬ 
ers  that  may  result  from  acquisition  reform 
policy  initiatives.  Continued  research  is 
also  being  conducted  to  produce  a  model 


for  evaluation  of  actual  criticality  of  re¬ 
ported  CSFs.  It  is  anticipated  that  the  de¬ 
velopment  of  this  model  will  provide  a 
means  to  alleviate  the  need  to  assume  that 
all  managers  are  equally  skilled  in  their 
ability  to  identify  CSFs  and  engage  in  ef¬ 
fective  strategic  thinking. 
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ACQUISITION  REVIEW  QUARTERIY 
GUIDELINES  FOR  CONTRIBUTORS 


The  Acquisition  Review  Quarterly 
(ARQ)  is  a  scholarly  peer-reviewed  jour¬ 
nal  published  by  the  Defense  Acquisition 
University.  All  submissions  receive  a 
masked  review  to  ensure  impartial  evalu¬ 
ation. 


Submissions 


Submissions  are  welcomed  from  any¬ 
one  involved  in  the  Defense  acquisition 
process.  Defense  acquisition  is  defined  as 
the  conceptualization,  initiation,  design, 
development,  test,  contracting,  produc¬ 
tion,  deployment,  logistic  support,  modi¬ 
fication,  and  disposal  of  weapons  and 
other  systems,  supplies,  or  services  to  sat¬ 
isfy  Defense  Department  needs,  or  in¬ 
tended  for  use  in  support  of  military  mis¬ 
sions. 


Research  Articles 


Manuscripts  should  reflect  research  or 
empirically-supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research  or  tutorial  articles 
should  not  exceed  4,500  words.  Opinion 
pieces  should  be  limited  to  1,500  words. 

We  publish  Defense  Acquisition  re¬ 
search  articles  that  involve  systemic  in¬ 


quiry  into  a  significant  research  question. 
The  article  must  produce  a  new  or  revised 
theory  of  interest  to  the  acquisition  com¬ 
munity.  You  must  use  a  reliable,  valid  in¬ 
strument  to  provide  your  measured  out¬ 
comes. 


Manuscript  SEaioNS 


The  introduction  should  state  the  pur¬ 
pose  of  the  article  and  concisely  summa¬ 
rize  the  rationale  for  the  undertaking. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly  de¬ 
scribes  work  performed.  Although  it  is 
appropriate  to  refer  to  previous  publica¬ 
tions  in  this  section,  the  author  should  pro¬ 
vide  enough  information  so  that  the  expe¬ 
rienced  reader  need  not  read  earlier  works 
to  gain  understanding  of  the  methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 
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RESEARCH  COHSIPERATIONS 

Contributors  should  also  consider  the 
following  questions  in  reviewing  their  re¬ 
search-based  articles  prior  to  submission: 

•  Is  the  research  question  significant? 

•  Are  research  instruments  reliable  and 
valid? 

•  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  under 
study? 

•  Does  the  research  design  fully  and  un¬ 
ambiguously  test  the  hypothesis? 

•  Did  you  build  needed  controls  into  the 
study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 
any  materials  and  methodology  necessary 
to  verify  their  conclusions. 


CritcriaforTutoriaes _ _ 

Tutorials  should  provide  special  in- 
straction  or  knowledge  relevant  to  an  area 
of  defense  acquisition  to  inform  the  De¬ 
fense  Acquisition  Workforce. 

Topics  for  submissions  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theory.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose. 


Opinion  Criteria 


Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and  pre¬ 
sented  in  a  factual  manner;  that  is,  sub¬ 
missions  should  imply  detachment.  The 
observation  and  judgment  should  not  re¬ 
flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  opinion  pieces 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 


Manuscript  Style 


We  will  require  you  to  recast  your  last 
version  of  the  manuscript,  especially  ci¬ 
tations  (e.g.,  footnotes  or  endnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the  Ameri¬ 
can  Psychological  Association  (4th  Edi¬ 
tion),  and  the  Chicago  Manual  of  Style 
(14th  Edition).  The  ARQ  follows  the  au¬ 
thor  (date)  form  of  citation. 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents. 
Standard  formulas  of  citations  may  give 
only  incomplete  information  in  reference 
to  government  works.  Helpful  guidance 
is  also  available  in  Gamer,  D.L.  and  Smith, 
D.H.,  1993,  The  Complete  Guide  to  Cit¬ 
ing  Government  Documents:  A  Manual 
for  Writers  and  Librarians  (Rev.  Ed.), 
Bethesda,  MD:  Congressional  Informa¬ 
tion  Service,  Inc. 
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Copyright  iNFORMAnoH 


The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Contributors  of  copyrighted 
works  and  copyright  holders  of  works  for 
hire  are  strongly  encouraged  to  request 
that  a  copyright  notification  be  placed  on 
their  published  work  as  a  safeguard  against 
unintentional  infringement.  The  work  of 
federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copy¬ 
right. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 
will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  editor  before 
publication. 


Manuscript  Format 


Pages  should  be  double-spaced  and  or¬ 
ganized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note 
(if  any),  and  figures  or  tables.  To  ensure 
anonymity,  each  paper  should  be  submit¬ 
ted  with  a  separate  page  that  includes  the 
author(s)’s  name(s)  and  complete  address, 
and  the  paper  should  include  the  title,  ab¬ 
stract,  keywords,  body,  complete  set  of 
references,  along  with  tables  and  figures 
at  the  end.  Authors  are  reminded  not  to 
refer  to  themselves  or  to  their  own  work 
directly  in  the  paper.  Figures  or  tables 
should  not  be  inserted  (or  embedded,  etc.) 


into  the  text,  but  segregated  one  to  a  page 
following  the  text.  Articles  must  be  print¬ 
able  within  one  issue  and  should  not  ex¬ 
ceed  4,500  words  for  research  or  tutorials 
and  1,500  words  for  opinion  pieces;  ar¬ 
ticles  will  not  be  printed  in  parts  or  in  a 
continuing  series.  If  material  is  submitted 
on  a  computer  diskette,  each  figure  or  table 
should  be  recorded  in  a  separate,  export¬ 
able  file  (i.e.,  a  readable  .eps  file).  For 
additional  information  on  the  preparation 
of  figures  or  tables,  see  CBE  Scientific 
Illustration  Committee,  1988,  Illustrating 
Science:  Standards  for  Publication, 
Bethesda,  MD:  Council  of  Biology  Edi¬ 
tors,  Inc.  Please  restructure  briefing  charts 
and  slides  to  a  look  similar  to  those  in  pre¬ 
vious  issues  of  ARQ. 

The  author  (or  corresponding  author  in 
the  case  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  the  author’s  name,  ad¬ 
dress,  and  telephone  number  (fax  and 
Internet  addresses  are  also  appreciated). 
The  letter  should  verify  that  the  submis¬ 
sion  is  an  original  product  of  the  author; 
that  it  has  not  been  published  before;  and 
that  it  is  not  under  consideration  by  an¬ 
other  publication.  Details  about  the  manu¬ 
script  should  also  be  included  in  this  let¬ 
ter:  for  example,  its  title,  word  length,  the 
need  for  copyright  notification,  the  iden¬ 
tification  of  copyrighted  material  for 
which  permission  must  be  obtained,  a  de¬ 
scription  of  the  computer  application  pro¬ 
grams  and  file  names  used  on  enclosed 
diskettes,  etc. 

The  letter,  one  copy  of  the  printed 
manuscript,  and  any  diskettes  should  be 
sturdily  packaged  and  mailed  to:  Defense 
Systems  Management  College,  Attn: 
DSMC  Press  (ARQ),  9820  Belvoir  Road, 
Suite  3,  Fort  Belvoir,  VA  22060-5565. 
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In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be  re¬ 
ferred  to  referees  and  subsequent  consid¬ 
eration  by  the  ARQ  Editorial  Board. 


Contributors  may  direct  their  questions 
to  the  Editor,  ARQ,  at  the  address  shown 
above,  by  calling  (703)  805-4290  (fax 
805-  2917),  or  via  the  Internet  at; 
gonzalezd  @  dsmc  .dsm.mil. 

The  DSMC  Home  Page  can  be  accessed 
at: 

http://www.dsmc.dsm.mil. 
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